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APPENDIX A: MORE ON THE TEXT MESSAGING STRATEGY 


Rationale for the Adaptive Text Messaging Strategy 


The adaptive text messaging strategy was meant to address districts’ need to reduce chronic absenteeism (Bauer, 
Liu, Whitmore Schanzenbach, & Shambaugh, 2018), often working with limited resources. The strategy built on 
prior research that suggested a low-cost approach - providing parents with basic information about attendance - 
could work, but tested the idea that messaging about attendance may be more effective if it adapted over time to 
add more information and motivation for those who need it. As described in the sections that follow, the study 
was designed to answer questions about the feasibility and effectiveness of text messaging about attendance, the 
benefit of using an adaptive approach, and how to best convey attendance-related information. 


Research has found that lower-cost approaches can improve attendance. For example, studies have found that 
periodic paper-based mailings that inform parents about the general importance of attendance and provide 
specific information about children’s attendance can reduce absences in grades 1-12. One study found a single 
mailing reduced average absences by 0.13 days over one semester.! Another study found that up to 5 mailings 
reduced average absences by 1 day over two semesters.” However, research on the effectiveness of texting about 
attendance - another potentially low-cost approach - is limited, and result have been mixed. One study texted 
parents of high school students in New York City over one semester and observed no impact on attendance.’ A 
second study texted parents of kindergarteners in one school in Pittsburgh and saw initial promising results.* No 
studies had examined the impact of texting parents in all elementary grades across whole school districts. 


Despite the limited research, texting is of particular interest to districts and others interested in improving 
education outcomes using low-cost approaches. Unlike mailings, texts can reach most parents quickly; cell 
phone ownership is high, and most received texts are read within minutes.* Additionally, current technology 
makes it possible to easily identify parents that need more support and adapt texting to try and better meet their 
needs. For example, text messages can be used to easily initiate two-way interaction with parents or to offer 
customized information addressing particular challenges. 


This study tested an adaptive text messaging strategy that was informed by the attendance-focused mailings 
studies just discussed, research on texting about other aspects of education (for example, college enrollment), 
and by research in other fields. For example, like the Rogers et al. studies, the adaptive text messaging strategy 
presented both general information about the importance of attendance and specific information about parents’ 
own children’s absences. The texts also drew on research suggesting that behaviors can be changed with the use 
of planning prompts and goal commitment.*® The study also drew on evidence from public health and prevention 
about using texting to change behavior. Specifically, as done in public health, the study used an “information- 


‘Rogers, Duncan, Wolford, Ternovski, Subramanyam, & Reitano, 2017. 

2 Rogers & Feller, 2018. 

3 Balu, Porter, & Gunton, 2016. 

4 Smythe-Leistico & Page, 2018. 

5 In 2018, more than 95 percent of the U.S. adult population owned cell phones, and more than 77 percent owned smart 
phones (http://www.pewinternet.org/fact-sheet/mobile/). In 2017, one-in-five adults whose annual household income fell 
below $30,000 were smartphone-only Internet users (http://www.pewresearch.org/fact-tank/2017/06/28/10-facts-about- 
smartphones/). In addition, 95 percent of business text messages are read within 5 minutes of being sent 
(https://www.textrequest.com/blog/complete-overview-business-texting). 

§ Castleman, 2015; Mayer, Kalil, Oreopoulous, & Gallegos, 2015; York & Loeb, 2014. See Exhibit A.2 for more detailed 
information about the research findings that informed the components of the adaptive text messaging strategy. 
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motivation-behavioral skills (IMB) model”’ to identify factors related to elementary school students’ attendance 
that could potentially be changed by texting parents. 


Based on the IMB model, the study identified three main factors that text messaging might address: (1) 
information deficit (for example, a lack of adequate information about the consequences of chronic absence or 
not knowing how many days a child has missed school), (2) lack of personal or social motivation (for example, 
not believing school will make a difference in the child’s future), and (3) lack of behavioral skills (for example, 
not using or knowing effective strategies to help children get to school on time, such as making sure to set an 
alarm) (see Exhibit A.1). According to the logic model for the study, changes in parents’ beliefs, motivations, and 
behaviors should improve student attendance and reduce chronic absence. Then, more time in the classroom 
because of regular attendance should theoretically improve students’ academic achievement outcomes because 
they are in school more and have more opportunities to learn. 


Exhibit A.1. Logic model for the study’s attendance text messaging strategy 


Attendance Text Messaging Strategy 


° iJaeyVilel=Xmlalielanatcharelammenrelanvcclilelan 
and helpful tips/skills 
Initially low cost, moves to higher 
cost as necessary 


Malleable Causes of 
Student Absenteeism 


Lack of Information 
About Consequences 
of Absenteeism 


Parent Outcomes Student Outcomes 


Parent 
Lack of Motivation Beliefs and 
Behavior 


Student 
Achievement 


Student 
Attendance 


You ao) S).411 53 


Description of the Adaptive Text Messaging Strategy 


The adaptive text messaging strategy had two phases. The first phase began in October 2017, when families were 
randomly assigned to one of two basic messaging approaches, Benefits-Framed Basic Messaging or Consequences- 
Framed Basic Messaging, or to the no-messaging control group. The second phase began in January 2018, when 
families whose children were frequently absent in the fall - despite the fall messaging - were randomly assigned 
to one of two intensified messaging approaches: School Staff Outreach or Goal Commitment messaging. Intensified 
messaging was in addition to continued basic messaging for the rest of the school year. Children were considered 
“frequently absent” if they missed 8 percent or more of school days in the fall, between October 1 and the end of 


7 Fisher & Fisher, 1992. 


ll 


December. Parents whose children missed fewer than 8 percent of school days continued to receive just the 
basic messages for the rest of the school year.*® 


The study focused on elementary aged students (grades K-5) because chronic absence in the early grades is an 
ongoing problem, and the impact of texting about attendance had not been previously examined at these grade 
levels. Almost 4 million elementary school students were chronically absent during the 2015-2016 school year, 
and missing this much school in early grades is linked with lower reading and math achievement by Grade 3 and 
higher absenteeism in middle and high school.® The messages that were part of the adaptive texting strategy, 
described next, were expressly designed for parents of elementary school aged children. 


Basic Messaging 


The purpose of the basic messaging was to communicate with parents about the importance of school 
attendance and to provide information about their own children’s absences in more meaningful and 
personalized ways than existing attendance communications between schools and parents. When 
communicating this type of information, a school district or provider needs to consider the tone or framing of 
the messages. Some evidence suggests that framing in terms of potential loss is more effective than framing in 
terms of potential gain, © but it was not clear if this would hold true in the context of texting parents about 
attendance. The study messages were designed to test whether it is better to frame attendance-related 
information in terms of positive benefits of consistent attendance, or negative consequences of absenteeism." 


Benefits-Framed Basic Messaging included (a) preventive weekly messages sent on Sundays, and (b) same-day 
notifications to parents on school days when their child was absent, and when their child was present after a 
prior-day absence. 


a. Weekly preventive messaging. Parents of all students in the benefits-framed basic messaging group received an 
automated message on Sunday nights about the importance and positive benefits of consistent attendance, 
with “tips” about avoiding common reasons for absences that included links to additional resources on 
external websites. 


8 An 8 percent absence rate, instead of the 10 percent commonly used to identify students as chronically absent, was chosen 
as the criterion for intensified messaging because absence rates can increase over the year. The study team erred on the side 
of casting a wider net for identifying “at risk” students rather than potentially missing some who might become chronically 
absent later in the year. 

° Applied Survey Research, 2011; Ehrlich et al., 2014; Ready, 2010. 

10 Castleman, 2015. 

4 Other investigations have tested the impact of providing personalized information relative to social norms, for example, in 
energy usage (Allcott, 2011). In their study of attendance-related mailings for parents of students in grades 1-12, Rogers and 
Feller (2018) provided attendance information with and without absolute or relative norms. While the study found that the 
mailings reduced absences overall, the use of social norms did not have an added benefit. 

As described in this section, basic messaging had two components: general weekly preventive messaging and personalized 
same-day notifications. The study focused on testing two different approaches to framing basic messaging, not parsing out 
the effects of general weekly preventive messaging and personalized same-day notifications. 

5 Resources used for the content of the tips messages were drawn from national and local websites and materials that 
covered topics such as the school calendar and information about the district’s attendance policy; how to get children ready 
for school in the morning; transportation (e.g., schedule to public transportation, importance of backup plans, how to report 
late/no-show buses); homework completion and resources for getting help with homework; health-related information, 
including influenza and vaccinations, allergies, toothaches, and how to decide when a child is too ill to go to school; bullying; 
and school anxiety/avoidance. 
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b. Same-day notifications. When a child was recorded as absent by the school in the district’s student 
information system, an automated text message was sent to the parents notifying them of the absence. These 
same-day notifications were personalized with the child’s name and included the number of days absent so 
far during the school year and a positively stated benefit of regular attendance. If a child missed four 
consecutive days of school, parents received a text informing them that the same-day notifications would 
end and asked them to contact the child’s school about their child’s absence if they had not done so already. 
When a child was marked absent the day before and then marked present on the current day, a “welcome 
back” message was sent to parents to acknowledge their child’s attendance. 


Consequences-Framed Basic Messaging also included weekly preventive messages, and same-day notifications 
to parents when their child was absent and when their child was present after a prior-day absence. 


a. Weekly preventive messaging. The Sunday automated messages for consequences-framed basic messaging 
noted the importance of regular attendance and consequences of chronic absenteeism and, like benefits- 
framed basic messaging, included tips with links to additional resources. 

b. Same-day notifications. As with benefits-framed basic messaging these automated messages were 
personalized with the child’s name and included the number of days absent so far, but instead of a positively 
stated benefit of regular attendance, the message included a briefly stated negative consequence of missing 
school. As with benefits-framed basic messaging, parents whose child was absent four consecutive days were 
informed that the same-day notifications would end, and when a child was marked absent one day and 
present the next, the parent was sent a message to acknowledge their child’s return to school. 


Intensified Messaging 


The two intensified messaging approaches (January-June) were (1) School Staff Outreach and (2) Goal Commitment 
messaging. Both intensified messaging approaches were intended to offer more tailored information and 
resources to parents and to increase parent motivation to make sure their child attends school regularly. The 
two intensified approaches reflect common messaging strategies that have some promising evidence for 
changing behavior but had not been successful in all studies and had not been tested in education: increased 
personalization with more tailored information provided by an identified contact person (School Staff Outreach) 
and providing more tailored information together with goal commitment (Goal Commitment). 


School Staff Outreach involved school staff attempting to initiate contact with parents through text messaging. 
The School Staff Outreach approach aimed to (a) increase parents’ motivation to make sure their child attends 
school each day through person-to-person contact with the school, and (b) improve their behavioral skills related 
to getting their child to school each day through tailored support and resources that address their specific needs 
and circumstances. 


School leaders and a district coordinator identified a staff person in each school to conduct the outreach. School 
staff were trained by the study team to conduct outreach in December 2017, with additional booster training in 
January 2018. The training provided information about how to access the platform, send and reply to text 
messages, and update parent contact information. School staff members were instructed to attempt to contact 
parents of all students who were assigned to the School Staff Outreach condition once a week for four weeks 
starting in mid-to-late January and once a month thereafter if parents did not respond within the first four weeks 
of outreach.'* When parents responded to the school staff member, school staff were encouraged to continue to 
engage with parents through texting or to communicate with parents through other means (for example, phone, 


“ The start time for the School Staff Outreach approach varied by district schedule and weather-related school closings during 
the first two weeks of January. 
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e-mail, in person) to better understand the underlying reasons for student absences and to refer parents to 
school, district, or community supports. 


Goal Commitment messaging entailed weekly goal-setting messages and options for obtaining more tailored tips 
and resources. Goal Commitment messaging aimed to (a) increase parents’ motivation to ensure their child’s 
attendance through weekly goal-setting with weekly feedback, and (b) improve their behavioral skills related to 
their child’s attendance with the addition of options to receive more tips or resources on topics of relevance to 
them. 


Parents in this group were sent a text message on Sundays asking them to commit to a goal of perfect attendance 
for the upcoming week. This message was interactive, requesting that parents reply ‘yes’ to commit to the goal. 
On Fridays parents who had replied ‘yes’ or did not reply were sent a message stating whether they had met the 
week’s goal of perfect attendance. Parent who had responded ‘no’ did not receive the Friday “feedback” 
message. When the child met the weekly attendance goal, the feedback praised them for the perfect attendance; 
when the child missed the goal, the feedback encouraged them to meet the goal the following week. 


Parents in Goal Commitment messaging also were sent an interactive version of Sunday’s tip messages. These 
messages invited parents to request additional tips by texting back a code word. Parents were able to request a 
total of three additional tips for each week’s topic. 


The higher cost intensified messaging approaches were reserved for the parents of children who were above the 
high absence threshold despite basic messaging in the fall. While all of the messaging approaches that were part 
of the adaptive text messaging strategy were generally low-cost (relative to other types of school-based 
interventions that require teacher training, materials, technology, or other resources), the intensified messaging 
approaches were costlier than the basic messaging approaches. School Staff Outreach added direct outreach 
from school staff to parents and thus took more staff time and effort than automated text messages. Goal 
Commitment messaging, though automated, added two additional components that asked for more parent time 
and effort: weekly goal setting and options to obtain tailored tips and resources. In addition to cost, the basic and 
intensified messaging approaches differed in emphasis with respect to the information-motivation-behavioral 
skill model. Basic messaging was primarily informational, while both intensified messaging approaches aimed to 
increase parent motivation and behavioral skills related to their children’s attendance. 


Exhibit A.2 describes the different messaging approaches, their purpose, and existing evidence for why these 
messaging approaches might be effective. Exhibits A.3 to A.7 provide examples of the text messages that were 
sent to parents in the two basic messaging approaches (benefits-framed and consequences-framed) and two of 
the basic plus intensified messaging combinations in the spring (benefits-framed basic messaging with School 
Staff Outreach, and consequences-framed basic messaging with Goal Commitment messaging). 
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Exhibit A.2. Description of the messaging components, the purpose and evidence for each component, and example messages 


Component 


Description 


Purpose 


Benefits-framed or Consequences-framed basic messaging 


Research evidence 


Example messages 


Weekly All parents, regardless whether Inform parents about the An evidence base for informational Benefits-framed message: Regular 
informational their child was absent, importance of regular interventions is emerging in health and attendance at school helps children 
messages received benefits-framed or attendance or about the prevention (Perry, Chhatralia, Damesick, learn good study habits. 
consequences-framed importance of avoiding Hobden, & Volpe, 2015) and education Consequences-framed message: 
messages on Sunday absences [addressing (Bergman, 2014; Castleman & Page, 2014; Missing school makes it hard for 
afternoons about the information in the 2016; Kraft & Dougherty, 2013; Kraft & children to learn good study habits. 
importance of attendance. information-motivation- Rogers, 2015; Rogers & Feller, 2018). 
behavioral skills (IMB) Some evidence suggests that framing in 
model]. terms of potential loss is more effective than 
framing in terms of potential gain 
(Castleman, 2015), especially for health- 
related information. However, the effect of 
framing had not been tested in the context of 
student attendance communications. 
Weekly tip Provided parents information Provide parents “tips” about Planning prompts help people make the link Some kids want to skip school 
messages on Sunday afternoons about avoiding common reasons for between a future event (getting child to because of unfinished homework. 


how to overcome common 
reasons for absences in the 
form of easy-to-understand tips 
and links to additional 
resources (e.g., websites). 


absences; prevent absences 
due to issues parents can 
address themselves 
(addressing behavior in the 
IMB model). 


school) and an action they need to take 
(getting child up on time) (Castleman, 2015). 


For homework help ideas, go to 
[LINK]. 
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Component 


Same-day 
notifications 


Description 


Acknowledged child’s absence 
with a text sent that day, 
personalized with names. 
Provided parents a brief 
benefits-framed reason that 
attending school is important. 


Purpose Research evidence 


Being sent the message the Text messages operate as a notification that 


same day that the child tends to attract people’s attention 

misses school informs parents (Castleman, 2015). The just-in-time aspect of 
that the district monitors the same-day absence notifications and 
student attendance and updated absence information emphasizes 
considers it a priority that the information is important (Abroms et 
(addressing information in al., 2015). Message personalization also has 
the IMB model). been found to be effective in health (Abroms 


et al., 2015; Head, Noar, Iannarino & 
Harrington, 2013; Klasnja & Pratt, 2012; 
Militello, Kelly & Melnyk, 2012). 


Example messages 


Benefits-framed message: We missed 
[CHILD] in school today. [CHILD] 
has missed a total of [X] days of 
school this year. Going to school 
every day will help [CHILD] learn 
important reading and math skills. 
Consequences-framed message: We 
missed [CHILD] in school today. 
[CHILD] has missed a total of [X] 
days of school this year. Missing 
school makes it harder for [CHILD] 
to learn important reading and math 
skills. 


Stop texting rule 


If child was marked absent for 
four consecutive days, parents 
were sent a message informing 
them that their child has now 
missed four days in a row and 
that the texting will stop until 
the child has returned to 
school. 


The “stop texting” rule is 
intended to acknowledge that 
some students may have 
chronic illnesses and parents 
may become frustrated with 
daily messages. 


[CHILD] has missed 4 days of school 
in a row. If you have not contacted 
the school about the absences, 
please call [NUMBER] to notify your 
child's school. 


Positive feedback 
message 


Send parents positive 
notification when their child 
returns to school. 


Messages use positive 
reinforcement to keep 
parents reading the texts. 


We were glad to have [CHILD] back 
to school today. Daily attendance 
helps [CHILD] reach their full 
potential. 
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Component 


Description 


Intensified messaging: School Staff Outreach 


Purpose Research evidence 


Example messages 


School-staff- 
initiated, two-way 
communication 


A text message was sent from 
school staff to parents offering 
more personalized help. The 
text messages could have been 
followed up with other means 
of communication. If parents 
did not respond, school staff 
were expected to follow up 
weekly for one month and 
then monthly after that. Staff 
were expected to provide 
information about the types of 
supports the school and 
district could provide and link 
parents to the supports as 
needed. 


Increase parent motivation The evidence on supplemental in-person 
through personal attention; |§ communication in health research is mixed, 
increase parent awareness of with some support for benefits of in-person 
local resources; offer more communication. Studies have found that the 
tailored support to address identity of the provider of the information is 


their child’s attendance important.’ Attention to the identity of the 
problems, by inviting provider of information is thought of as a 
communication in a non- “tailoring” strategy in the health fields 
invasive manner. (Abroms et al., 2015; Head, Noar, Iannarino & 


The goal was for the staffto | Harrington, 2013). (In this study the school 
identify the barriers to regular staff outreach messages were sent by a 
attendance and link families | named, identifiable person in the school, and 
with appropriate the messages provided an opportunity for 
supports/services. (The staff | two-way communication.) 

were provided an inventory 

of locally-available 

supports/services for 

different barriers, with 

contact information.) 

(Addressing information and 

motivation in the IMB model.) 


Dear [PARENT NAME], 

This is [name and position]. [CHILD] 
has missed [X] days of school so far 
this year. 

Please text me at [NUMBER] or call 
me at [NUMBER] so we can set up a 
time to talk. Iam here to help. 
Thank you 
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Component Description Purpose Research evidence 


Intensified messaging: Goal Commitment 


Example messages 


Parent Offered parents an opportunity Engage parents and Commitment devices have mixed support 
engagement: to commit to an attendance encourage them to commit to (Perry et al., 2015), but potential for broad 
interactive goal of zero absences for a an attendance goal and specific application to help people 
commitment school week. (addressing information and _ achieve goals (Castleman, 2015). Some 
device with Parents were sent a motivation in the IMB model). studies have found parent commitment or 
weekly summary commitment text at the engagement strategies to be effective for pre- 
beginning of each week. school children (Mayer, Kalil, Oreopoulos & 
Parents were asked to commit Gallegos, 2015; York & Loeb, 2014) but 
to the goal or not - parents commitment to attendance goals had not 
who committed via text and been tested with elementary school aged 
parents who did not commit children. 


received a summary report at 
the end of the week whether 
the attendance goal was met or 
not. 


The end-of-week summary 
provided additional 
encouragement and 
functioned as a reward for the 
interactive commitment 
device. 


Weekly invitation to commit 
message: 

Children lose opportunities to learn 
when they miss school. 

Will you make it a goal that [CHILD] 
attends school every day this week? 
Reply YES to join other parents in 
setting this goal. 

‘Met Goal’ feedback message: 
[CHILD] had perfect attendance this 
week! Keep up the great work. 

‘Did Not Meet Goal’ feedback 
message: [CHILD] missed the weekly 
attendance goal this week. Please 
help us meet this goal next week! 


4 However, even with mixed results, including two-way communication as part of the messaging strategy was important for the following reasons: 


Many, if not all, schools and districts provide in-person supports and resources for parents with chronically absent children, and it was important that the text messaging strategy 
aligned and did not conflict with existing supports that are more intensive. Many districts have already adopted a multi-tiered system of support for promoting attendance. The two- 


way communication option could help bridge parents to supports that are more intensive. 


Not all parents will receive the text messages (e.g., they may not have paid their phone bills, the phone may be broken, or they may have changed numbers). Providing an option for 


reaching out to parents who are not responsive to the basic text messaging approaches may increase the effectiveness of the strategy. 
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Exhibit A.3. Benefits-framed basic messaging (two hypothetical weeks in fall 2017) 


Day Event Message 
First week 
Sunday Informational Children who attend school every day or miss at most 1 day a month, are more likely to 
message read on or above grade level. 

Some kids want to skip school because of unfinished homework. For homework help 
ideas, go to [LINK] 

Monday NA 

Tuesday Me 

Wednesday NA 

Thursday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Going to school every day will help [CHILD] learn important reading and 
math skills. 

Friday NA 

Second week 

Sunday Informational Going to school regularly helps children learn how to finish the tasks they start. 

message Some children are hard to get out of bed in the morning. For more information, click 

[LINK]. 

Monday NA 

Tuesday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Going to school every day will help [CHILD] learn important study skills. 

Wednesday Student returns We were glad to have [CHILD] back to school today. Daily attendance helps [CHILD] 
learn and succeed. 

Thursday NA 

Friday NA 
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Exhibit A.4. Consequences-framed basic messaging (two hypothetical weeks in fall 2017) 


Day Event Message 
First week 
Sunday Informational Attendance Matters! Students who miss 2 or more school days a month are less likely to 
message read on or above grade level. 

For more information about the importance of attendance and district attendance 
policies, go to [LINK]. 

Monday NA 

Tuesday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Missing school makes it harder for [CHILD] to learn how to work well with 
peers. 

Wednesday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Missing school means [CHILD] cannot participate in class activities. 

Thursday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Missing school makes it harder for [CHILD] to get good grades. 

Friday Student absent [CHILD] has missed 4 days of school in a row. If you have not contacted the school 
about the absences, please call [NAME] at [NUMBER]. 

Second week 

Sunday Informational Missing school makes it hard for children to learn good study habits. 

message Some students may get stomachaches because they are worried or nervous about 

school. For more information, go to [LINK]. 

Monday NA We were glad to have [CHILD] back to school today. Daily attendance helps [CHILD] 
learn the skills needed to move on to the next grade level. 

Tuesday NA 

Wednesday NA 

Thursday Ne 

Friday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 


this year. Missing school makes it harder for [CHILD] to learn the skills needed to move 
on to the next grade. 
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Exhibit A.5. Benefits-framed basic messaging with School Staff Outreach (two hypothetical weeks in 


spring 2018) 
Day Event Message 
First week 
Sunday Informational Regular school attendance is one of the biggest factors that can help a child’s academic 
message success. 


Some children have a hard time transitioning from home to school. For more 
information, go to [LINK]. 


Student absent 


We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 


Monday 
this year. Going to school every day will help [CHILD] participate in class activities. 
Tuesday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Going to school every day will help [CHILD] learn the skills needed to move 
to the next grade level. 
Wednesday School Staff Dear [PARENT NAME], 
Outreach This is [name and position]. [CHILD] has missed [X] days of school so far this year. 
Please text me at [NUMBER] or call me at [NUMBER] so we can set up a time to talk. I 
am here to help. Thank you. 
Thursday Student returns We were glad to have [CHILD] back to school today. Daily attendance helps [CHILD] 
reach their full potential. 
Friday a 
Second week 
Sunday Informational Children who attend school every day or miss at most 1 day a month, are more likely to 
message graduate from high school. 
It’s important to know the school calendar. Visit [LINK] before planning family events 
or travel. 
Monday NA 
Tuesday Nes 
Wednesday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Going to school every day will help [CHILD] learn important study skills. 
Thursday Student returns We were glad to have [CHILD] back to school today. Daily attendance helps [CHILD] 
learn and succeed. 
Friday School Staff Dear [PARENT NAME], 
Outreach 


This is [name and position], trying again to reach you. [CHILD] has missed [X] days of 
school so far this year. Please text me at [NUMBER] or call me at [NUMBER] so we can 
set up a time to talk. 1am here to help! 
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Exhibit A.6. Consequences-framed basic messaging with Goal Commitment messaging (two 
hypothetical weeks in spring 2018) 


Day Event Message 
First week 
Sunday Informational It can be hard to get your child to school on time. Text “TIP” for tips on how to get out 
message and Goal the door in the morning. 
Commitment Children lose opportunities to learn when they miss school. Will you make it a goal that 
message [CHILD] attends school every day this week? Reply YES to join other parents in setting 
this goal. 
Monday NA 
Tuesday Be 
Wednesday NA 
Thursday Ne 
Friday Weekly [CHILD] had perfect attendance this week! Keep up the great work 
attendance 
feedback 
Second week 
Sunday Informational Spring allergies can trigger asthma in some kids. Text “TIP” for ways to keep asthma 
message and Goal under control. 
Commitment Children lose opportunities to learn when they miss school. Will you make it a goal that 
message [CHILD] attends school every day this week? Reply YES to join other parents in setting 
this goal. 
Monday ae 
Tuesday NA 
Wednesday NA 
Thursday Student absent We missed [CHILD] in school today. [CHILD] has missed a total of [X] days of school 
this year. Missing school makes it harder for [CHILD] to feel like an important part of 
the school. 
Friday Student returns, We were glad to have [CHILD] back to school today. Daily attendance helps [CHILD] 


Weekly 
attendance 
feedback 


reach their full potential. 


[CHILD] missed the weekly attendance goal this week. Please help us meet this goal 
next week. 
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Exhibit A.7. Example of interactive messages that were part of the Goal Commitment messaging 
strategy 


Give your child a picture or note 

id utclmau(yymetou Cole) @-tmeltlulerem iets 
day to remind them of home. Text 
"tipAnxiety2" for more ideas. 


Develop a goodbye ritual like a 


special wave. Rituals can help kids 


feel more secure. For more 
helpful tips, text "tipAnxiety3." 


Monitor your child’s TV and 
Internet use. Scary content —even 
the news —can make some kids 
anxious. For more, go to 


https://goo.gl/kAxkwn 


Development of the Adaptive Text Messaging Strategy 


Message Content Development and Usability Testing 


The study team developed the messaging content by drawing on behavioral science research and literature on 
the relationship between absences and later academic and social outcomes. To develop the content for the 
weekly tip messages, the study team identified national and local websites and resources. In addition, some 
messages provided parents direct access to these resources via links embedded in the texts. 


As part of the development process, the study team gathered parent feedback about the messages by conducting 
three rounds of focus groups. The study team recruited parents who had children in elementary grades from a 
large urban school district. During the usability testing sessions, study team members sent the text messages to 
parents’ cell phones to gather information on how the texts would appear and parent opinions about the 
messages. The first two rounds of usability testing focused on the tone, clarity, and length of the messages as well 
as initial impressions of the different messaging approaches. All parents that participated in the first two rounds 
of usability testing indicated that they did not want ‘cheerful,’ ‘cute,’ or ‘playful’ language, but rather preferred 
short, factual messages. The study team revised the content and tone of the messages after the first two rounds 
of feedback. The third round of usability testing focused on ‘acceptability,’ whether parents would find specific 
messaging approaches or message components (e.g., tips, same-day notifications, interactive goal commitment) 
off-putting. Most parents indicated that they found the same-day notifications that informed them about their 
child’s number of absences to be most useful. Parents did not have a negative reaction to the informational 
messages (e.g., pre-scheduled messages with tips). Most parents noted that some of the messages would not be 
relevant to their personal circumstances, but also acknowledged that they had friends or knew parents who 
could benefit from the information. 
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Translation of Messages to Different Languages 


The text messages were translated to multiple languages based on district requirements, including Amharic, 
Arabic, Chinese, French, Korean, Russian, Spanish, Somali, and Vietnamese. The text messages were 
professionally translated and also proofread and checked for clarity by study team staff who were native 
speakers of the languages. 


The home language for each student was recorded in the district’s student information system. Exhibit A.8 
shows the percentage of students in the study who had each language recorded as their primary language 
spoken at home. The text messaging platform used the information in the student information system to send 
messages in the recorded home language. If parents updated the home language information, for example from 
English to Spanish, the text messaging platform would automatically adjust the language of the sent messages to 
match the updated information. 


Exhibit A.8. Percentage of students in each messaging group with different primary languages spoken 
at home 


Students assigned to 


Students assigned to intensified All students in 

Students assigned to | intensified messaging: messaging: messaging 
Home language basic messaging only | School Staff Outreach Goal Commitment conditions 
Amharic 0.0 0.0 0.0 0.1 
Arabic 0.6 0.3 0.0 0.2 
Chinese 0.0 0.0 0.4 0.1 
English 75.3 80.2 83.1 80.3 
French 0.1 0.2 0.0 0.1 
Russian 0.1 0.0 1.2 0.4 
Somali 0.7 0.0 1.0 0.4 
Spanish 22.9 19.0 12.6 17.7 
Vietnamese 0.5 0.3 1.6 0.7 


SOURCE: District administrative records. 

NOTES: The analytic sample included students who were enrolled for 100 percent of the school days and had 15,269 students; 11,097 
students assigned to basic messaging only; 2,115 students assigned to basic messaging with School Staff Outreach, and 2,057 students assigned 
to basic messaging with Goal Commitment. Although one district requested to have the messages translated into Korean, there were no 
students in the study sample whose home language was indicated as Korean in the districts’ student information systems. 


District Approval 


Prior to starting any text messaging in fall 2017, district staff reviewed the messages to approve their content. The 
study team also requested district staff confirm that the messages related to school calendars were accurate. All 
four participating districts approved the messaging content without significant changes. Changes requested by 
districts included adding a district-specific attendance slogan into the messages, if they had one (such as 
“Attendance Matters!”), and specific changes to school district calendar information (such as the length of the 
winter break). 
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Description of the Text Messaging Platform 


The study text messages were sent out to parents in participating districts from a platform provided by a text 
messaging vendor. The study team selected the vendor, SchoolMessenger (Intrado), through a competition. The 
SchoolMessenger system allowed pre-scheduled messages to be uploaded into the platform by study team staff, 
who set the specific times for messages to be sent out. The system also allowed automated “conditional” 
messaging, meaning that whether a message was sent was triggered by the combination of daily attendance data 
(whether a student was in school or not) together with the study condition indicators. (That is, whether a student 
was assigned to benefits- or consequences-framed basic messaging and whether a student was also assigned to 
School Staff Outreach or Goal Commitment intensified messaging.) SchoolMessenger worked with the study team 
and each of the participating districts to establish procedures for transferring districts’ daily attendance data into 
the SchoolMessenger system so that the messages could be automatically sent according to the study design. 


The study team worked with SchoolMessenger to develop an additional platform module for the school 
attendance counselors to use for personalized outreach. The platform module limited school counselors to 
texting only the families of those students who were part of the School Staff Outreach group, and allowed school 
counselors to use templates to send out individual messages to multiple parents simultaneously as well as to 
customize their messages. The platform module could be accessed from a desktop, laptop, or iPad and aimed to 
be easy to use to reduce the need for extensive training. 


For the text messages to be sent out successfully, districts needed to have a student information system into 
which daily attendance was entered and that had the capacity to transfer daily attendance data to the text 
messaging platform. The transfer process involved correctly programming data queries that would create 
attendance variables needed for the daily attendance notifications, and then send the created dataset daily to the 
messaging platform on time to trigger attendance notifications. All districts in the study had a student 
information system that was updated daily with attendance information and a data analyst who facilitated the 
programming of queries and transferring data. 


The most common reasons messages were not sent or incorrect messages were sent were (a) errors in district 
programmed data queries that would define the attendance data variables used to trigger same-day absence 
notifications, (b) failure of the data queries to successfully create the dataset and send it to the text messaging 
platform on time, and (c) inadequate capacity of the text messaging system to send all the messages at the same 
time which caused some of the messages to not be delivered within the timeframe set by the study (not past 8 
pm local time). The programming errors in data queries and the capacity problem of the text messaging platform 
were generally limited to the first two weeks of the study. The district data analyst re-programmed queries as 
needed, and the limited message sending capacity was corrected by SchoolMessenger by adding more server 
capacity and by staggering sending of pre-programmed messages. Problems with data queries not creating the 
daily datasets on time persisted on and off throughout the year in one of the four participating districts. 


Identifying and Training School Staff to Implement School Staff Outreach 


District and school contacts identified a school staff person to conduct the School Staff Outreach. In one district, 
the outreach for students in multiple schools was conducted by district-level attendance analysts. In the other 
three districts, one or more school-based staff in each school conducted the outreach. Exhibit A.9 shows the 
number and percent of different types of school personnel who conducted the outreach in the study schools. 
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Exhibit A.9. Number and percentage of school staff in outreach position, by role 


Roles N Percent 
District attendance counselors 6 5.5 
School attendance counselors 7 6.4 
Parent liaisons 13 11.9 
School social workers/guidance counselors 33 30.3 
Other school staff 50 45.9 


SOURCE: School Messenger School Staff Outreach Accounts. 
NOTES: Sample size = 109 school staff members assigned to conduct School Staff Outreach. 
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APPENDIX B: MORE ON THE STUDY DESIGN 


Overview of the Design 


The evaluation used a sequential multiple assignment randomized trial (SMART) design, depicted in Exhibit B.1." 
SMART designs are used to compare adaptive interventions that tailor the type, dosage, or timing of an 
intervention based on study participants’ characteristics or initial response to the intervention. This study used a 
SMART design to compare interventions that adapted based on students’ fall absences to see which intervention 
best improved student attendance. In late September 2017, the study team randomly assigned families (and their 
students) within elementary schools to one of the three initial messaging conditions: benefits-framed basic 
messaging or consequences-framed basic messaging, or to the no-messaging control condition. Between October 
1 and the end of the fall semester, families received messages consistent with their basic messaging condition. 
Depending on their child’s absence rate during the fall, they then either continued with their basic messaging for 
the rest of the year, or they received additional intensified messaging during the spring semester. Specifically, 
families whose child(ren) missed fewer than 8 percent of the school days between October and the end of 
December 2017 continued with their basic messaging from January 2018 through the end of the school year. 
Families with at least one child who missed 8 percent or more of school days during this same period in the fall 
despite basic messaging were re-randomized in January 2018 to one of the two intensified messaging conditions: 
School Staff Outreach or Goal Commitment messaging. These intensified messaging strategies were added on top 
of basic messaging (benefits-framed or consequences-framed), which continued through the end of the school 
year. The combinations of basic and intensified messaging form four adaptive interventions shown in Exhibits 
B.2 to B.5. Note the main study report refers to these adaptive interventions as “four versions of the adaptive text 
messaging strategy.” 


The two sequential randomizations in the SMART design allow for causal comparisons of (1) the basic messaging 
strategies to each other and the no-messaging control condition, (2) the two intensified messaging strategies to 
each other, and (3) the four adaptive interventions to the no-messaging control condition and each other. Only 
families with at least one child who met the eligibility condition based on the pre-set tailoring variable were 
included in the second randomization. A primer published by the Institute of Education Sciences (Nahum-Shani 
& Almirall, 2019) offers additional details on SMART designs in education. 


5 For more information about SMARTs, see Almirall, Nahum-Shani, Sherwood, & Murphy (2014); Murphy & Almirall (2009); 
and Nahum-Shani et al. (2012). 
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Exhibit B.1. Sequential multiple assignment randomized trial (SMART) design for testing four versions 
of the adaptive text messaging strategy 


OCTOBER TO DECEMBER JANUARY TO JUNE 


Missed 8 Percent 
or More in the Fall 


Continue Benefits-Framed 
Messaging 

Benefits-Framed 

Basic Messaging Benefits-Framed Messaging 

with School Staff Outreach 


Benefits-Framed Messaging 
with Goal Commitment 


Continue Consequences-Framed 
Messaging 


Consequences-Framed 
Basic Messaging Consequences-Framed Messaging 
with School Staff Outreach 


Consequences-Framed Messaging 
with Goal Commitment 


No Messaging — Control N/A Continue with No Messaging 


Exhibit B.2. Adaptive intervention #1 (AI-1): Benefits-framed basic messaging with School Staff 
Outreach 


OCTOBER TO DECEMBER JANUARY TO JUNE 


Missed 8 Percent 


or More in the Fall 


Continue Benefits-Framed 


Messaging 
Benefits-Framed 
Basic Messaging Benefits-Framed Messaging 
with School Staff Outreach 
Yes ce) 
Benefits-Framed Messaging 


with Goal Commitment 


Continue Consequences-Framed 
[No | Messaging 
Basic Messaging Consequences-Framed Messaging 
@ with School Staff Outreach 


Consequences-Framed Messaging 
with Goal Commitment 


Consequences-Framed 


No Messaging — Control N/A Continue with No Messaging 
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Exhibit B.3. Adaptive Intervention #2 (AI-2): Benefits-framed basic messaging with Goal Commitment 


messaging 
OCTOBER TO DECEMBER 


JANUARY TO JUNE 


Missed 8 Percent 


or More in the Fall 


Benefits-Framed 
Basic Messaging 


Consequences-Framed 


Basic Messaging 


[ No | 
cane 


No Messaging — Control N/A 


Continue Benefits-Framed 
Messaging 


Benefits-Framed Messaging 
with School Staff Outreach 


Benefits-Framed Messaging 
with Goal Commitment 


Continue Consequences-Framed 
Messaging 


Consequences-Framed Messaging 
with School Staff Outreach 


Consequences-Framed Messaging 
with Goal Commitment 


Continue with No Messaging 


Exhibit B.4. Adaptive intervention #3 (AI-3): Consequences-framed basic messaging with School Staff 


Outreach 


OCTOBER TO DECEMBER 


JANUARY TO JUNE 


Missed 8 Percent 
or More in the Fall 


Benefits-Framed 
Basic Messaging 


Consequences-Framed 


Basic Messaging 


No Messaging — Control 


29 


Continue Benefits-Framed 
Messaging 
Benefits-Framed Messaging 
with School Staff Outreach 
Benefits-Framed Messaging 
with Goal Commitment 


Continue Consequences-Framed 
Messaging 


Consequences-Framed Messaging 
with School Staff Outreach 


Consequences-Framed Messaging 
with Goal Commitment 


Continue with No Messaging 


Exhibit B.5. Adaptive intervention #4 (AI-4): Consequences-framed basic messaging with Goal 
Commitment messaging 


OCTOBER TO DECEMBER JANUARY TO JUNE 


Missed 8 Percent 
or More in the Fall 
Continue Benefits-Framed 
Messaging 
Benefits-Framed Messaging 
with School Staff Outreach 
=}—(*) 
Benefits-Framed Messaging 
with Goal Commitment 


Continue Consequences-Framed 
Messaging 


Benefits-Framed 
Basic Messaging 


Consequences-Framed 
Basic Messaging 


Consequences-Framed Messaging 
with School Staff Outreach 


Consequences-Framed Messaging 
with Goal Commitment 


No Messaging — Control 


N/A Continue with No Messaging 


NOTES: “R” refers to random assignment of families (and their students) to one of the messaging conditions. “Eligible for intensified messaging” 
was defined as a family having at least one child who missed 8 percent or more of school days between October 1 and the end of December. 


Description of the District and School Samples 


Recruitment 


The study established the following eligibility criteria for district and school participation: 


e The district had at least 10 low-performing" elementary schools with high levels of absenteeism (i.e., more 
than 15 percent of students chronically absent), since these were the types of districts most likely to benefit 
from a text messaging approach of the kind studied. 


e The district and schools did not systematically send text messages to parents about attendance during the 
2016-17 school year and were not planning to start systematic text messaging to parents about attendance 
during the 2017-18 school year, so as not to have multiple texting systems at the same time.” 


'6 The U.S. Department of Education was specifically interested in a study that would improve outcomes for disadvantaged students. 
Although the study did not set a strict participation eligibility threshold for academic performance based on proficiency rates or 
other achievement measures, the study team examined the relative achievement levels of the potentially eligible schools and 
districts to ensure that the study took place in sites that reflected the target population of elementary schools with high absenteeism 
and low academic performance. For example, the average percentage of students proficient on English/language arts state 
standardized assessments in 2016-17 in each of the four participating districts was 18 percent, 25 percent, 38 percent, and 46 percent. 

'” Systematic text messaging was defined as (1) text messaging about attendance that was conducted through a district parent 
notification system either by district or school staff to all parents who met preset criteria—for example, every parent whose child is 
absent gets a text the same day; all parents whose child has missed 5 days of school receive a text message; or (2) school staff sending 
text messages consistently to all parents who meet specific preset criteria by using their own cell phones (not the district system). 
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e The text messaging intervention provided by the study would supplement and not conflict with the 
district’s current approach to improving student attendance. 


e The district was willing to work with the designated text messaging provider. 


e The district was able to share attendance and other relevant data with the study team. 


The study team used U.S. Department of Education data (Common Core of Data, 2015-16 and the Office for Civil 
Rights data, 2013-14) to identify the pool of districts that had at least 10 low-performing elementary schools 
(grades K-5/6) with high levels of absenteeism. The study team first reached out to these districts via e-mail and 
phone calls. The study team then visited those districts that expressed sufficient interest and appeared to meet 
the eligibility criteria described above. The final selection of four districts was based on their ability to meet the 
eligibility criteria above and to accommodate required research and Institutional Review Board protocols. 


A total of 108 elementary schools in these four districts participated in the study. Exhibit B.6 provides descriptive 
statistics about the participating schools next to those of similarly sized school districts serving grades K-5. 


Exhibit B.6. Background characteristics of schools in the study sample, schools serving grades K-5 in 
similarly sized districts 


Schools in Schools serving grades K-5 in 
Characteristics study sample similarly-sized urban districts 


Urbanicity (percent) 
Urban 97.2 88.0* 
Suburban 2.8 9.0* 
Town 0.0 0.2 
Rural 0.0 2.5 
Title I status (percent) 89.8 80.5* 
Students eligible for free or reduced-price lunch (percent) 83.4 69.9* 


Racial/ethnic composition (percent) 


White, non-Hispanic 16.1 24.4* 
Black, non-Hispanic 47.5 25.6* 
Hispanic 29.9 38.8* 
Asian/Pacific, non-Hispanic 2.7 6.5* 
Other, non-Hispanic 3.8 4.7 
Total school enrollment, mean 448.1 552.3* 
Percent of students chronically absent 27.7 12.0* 


SOURCE: Common Core of Data 2015-16 school year, Office for Civil Rights Data for the 2013-14 School Year 

NOTES: Sample size = 108 schools in study sample, 8,193—9,379 schools serving grades K-5 in similarly-sized urban districts, with the number 
of schools varying across different characteristics due to data availability. The percentage of students eligible for free or reduced-price lunch 
in the study sample was based on only 83 of the 108 study schools because these data were not available in the Common Core of Data for one 
of the four study districts (which had 25 study schools). 

Percentage values for characteristics with multiple categories may not sum to 100 because of rounding. 

p-values are based on comparisons against schools in study sample using a one-sample test of proportions for measures of urbanicity and 
Title I status, and a one-sample t test for the other variables: * Statistically significant at p < .05. 
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Random Assignment and Analytic Sample 


Families (and their students) were randomly assigned to one of the two basic messaging approaches or the 
control group within schools with equal probability. Students were grouped in “families” to prevent parents with 
multiple children in grades K-5 from getting different types of messages for their different children. In the 
simplest case, students who shared the same last name, address, and phone number were grouped together. 
Students who shared the same phone number and address but had different last names were also grouped 
together. The study team hand-coded all cases where students shared the same address but did not share the 
same phone number or last name. Multiple families in one address location were typically linked to homeless 
shelters. 


In addition to blocking by school (i.e., grouping families and their students by school, and then conducting 
random assignment within schools), two other blocking variables were used to ensure that each of the three 
initial study conditions had equal proportions of families with baseline characteristics relevant for the study’s 
intervention: (1) whether the family had one or more children who had a history of high absences, and (2) 
whether the family had at least one valid cell phone number with short message service (SMS), which was 
necessary for receiving the study’s text messages. A family was considered to have had a history of high absences 
if they had one or more children who had (1) missed 10 percent or more instructional days in the 2016-17 school 
year or (2) missed 10 percent of instructional days between the start of the 2017-18 school year and September 15, 
2017. A family was identified as having at least one working SMS number based on records collected in August- 
September, when text messages were sent out to parents in participating districts and schools, informing them 
about the study and giving them the chance to opt out of participating prior to random assignment to messaging 
groups or the no-messaging group. 


The sample at the time of first random assignment included 36,706 students in grades K-5 nested within 26,843 
families."* The study randomly selected one child per family to be included in the analytic sample.’ Exhibit B.7 
shows the number of students in each initial study condition, overall and by district. By design, the conditions 
had similar proportions of families with one or more children with a history of high absences and with a working 
SMS number. 


Exhibit B.7 also shows the number of students in each initial study condition who were in the sample used to 
analyze the effectiveness of the texting strategy, a total of 23,133 students. Students were in the analytic sample if 
they were enrolled for 100 percent of the school year, had complete demographic information at baseline, and 
were randomly selected for inclusion in the analytic sample if the family had multiple children in grades K-5. The 
main analyses restricted the sample to students who were enrolled for 100 percent of the school days to avoid 
the problem of missing attendance data. Missing attendance data complicates the definition of the attendance 
outcome measures, including chronic absence (missing 10 percent or more of school days between October 1 and 


'8 Of the 108 participating schools, 68 included students in prekindergarten, and 19 schools included students in grades 6, 7, 
or 8. Families with students only in prekindergarten or only in grades 6-8 were included in the random assignment process, 
at the request of the districts; however, they were blocked separately from families with students in grades K-5 and were not 
included in the analyses. 

The study did not randomly assign individual children to messaging condition to avoid sending parents with more than one 
child in grades K-5 different types of messages for their different children. However, the study team randomly selected one 
child per family for the sample used to analyze the effectiveness of the texting strategy. Doing so simplified the analytic 
models used to assess the impact of the messaging because it allowed the study to avoid including a “family level” in the 
analytic models. Including a family level would have complicated estimation because most families in the study had only one 
child in the relevant grades. The selection of one child per family for the analytic sample occurred after the implementation 
of text messaging was completed. 
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the end of the school year). In addition, the complete baseline demographic data allows inclusion of the baseline 
characteristics as covariates in the analytic models, which increases the precision of the impact estimates. The 
study team conducted sensitivity analyses using different sample definitions and different missing data analysis 
approaches. The principal conclusions are consistent with the ones from the analytic sample used for main 
analyses. (See Appendix E for sensitivity analyses.) Students whose parents opted them out of the study after 
random assignment were excluded from the analytic sample altogether. (See next section on Attrition.) 


Exhibit B.7. Number of students (and their families) in each initial study condition at the time of first 
random assignment in late September 2017 (after selecting one child per family for the analysis) and 
number of students (and their families) in each initial study condition in the final analytic sample 


Number of students (families) by initial study condition 


Benefits-framed | Consequences-framed 


Control group 
1 RA 2,221 2,226 2,223 6,670 
Analytic 1,934 1,963 1,986 5,883 
2 RA 2,582 2,576 2,573 7,731 
Analytic 2,117 2,057 2,239 6,413 
3 RA 1,550 1,549 1,537 4,636 
Analytic 1,423 1,419 1,431 4,273 
4 RA 2,607 2,600 2,599 7,806 
Analytic 2,191 2,167 2,206 6,564 
All districts RA 8,960 8,951 8,932 26,843 
Analytic 7,665 7,606 7,862 23,133 


SOURCE: Study records. 
NOTES: “RA” is the sample at random assignment. “Analytic” is the analytic sample. 


After the basic messaging took place between October 1 and late December 2017, the study team identified 
families for random assignment to the intensified messaging approaches (School Staff Outreach or Goal 
Commitment messaging). Families were randomly assigned to intensified messaging if they had been randomly 
assigned to basic messaging for the fall (either benefits or consequences-framed) and then had at least one child 
who exceeded the 8 percent absence threshold. Families with multiple children were eligible for intensified 
messaging based on the child with the highest number of absences, regardless of whether this child was 
randomly selected for the analytic sample. This meant that in some cases, the student randomly selected for the 
analytic sample, after implementation of the intervention was complete, was not the student with the highest 
number of absences in the family. The second randomization was also blocked by school. Exhibit B.8 shows the 
number of students in each intensified messaging group at the time of the second random assignment (January 
2018), and the number of students in each intensified messaging group who were in the analytic sample. 
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Exhibit B.8. Number of students (and their families) in each intensified messaging group at the time of 
second random assignment in January 2018 (after determining eligibility for intensified messaging) and 
in each intensified messaging group in the final analytic sample 


Number of students/families 


Consequences- 


Benefits-framed | Benefits-framed | framed basic with} Consequences- 


basic with School | basic with Goal School Staff framed basic with 
Nicvime@litaner(onl Commitment Outreach Goal Commitment 
1 RA 233 234 227 222 916 
Analytic 210 219 223 217 869 
2 RA 336 336 332 330 1,334 
Analytic 302 293 299 278 1,172 
3 RA 410 406 394 394 1,604 
Analytic 343 308 313 319 1,283 
4 RA 253 253 258 246 1,004 
Analytic 219 214 207 209 849 
All RA 1,232 1,229 1,205 1,192 4,858 
Analytic 1,074 1,034 1,042 1,023 4,173 


SOURCE: Study records. 
NOTES: “RA” is the sample at random assignment, “Analytic” is the analytic sample. 


Attrition 


Attrition occurs when participants who are originally in the study are not included in the analysis due to missing 
outcome data or another reason. Attrition can cause problems in impact evaluations because generalizability of 
findings can be limited if certain types of students are not represented in the analysis. Attrition can also 
introduce bias in study results if different types of students are more likely to be missing from one messaging 
group than another. 


In this study, there were three reasons for attrition (students from families that were randomly assigned to basic 
messaging in September 2017 but were not included in the study’s final analytic sample): (1) if their family opted 
out of the study after randomization,”° (2) if the students moved out of the district during the school year, or (3) 
if the students did not have complete demographic information in the datasets provided by districts, which were 
needed for the analysis. If parents unsubscribed from the text messaging for a given phone number after texting 
began, their child was still included in the analytic sample because they received at least some exposure to the 
strategy, and it was possible another family member was still receiving texts. 


The overall attrition rate was 13.8 percent. Exhibit B.9 shows the attrition rates for students in the seven possible 
pathways in the study: benefits-framed basic messaging only, consequences-framed basic messaging only, 
benefits-framed basic messaging with School Staff Outreach, benefits-framed basic messaging with Goal 
Commitment, consequences-framed basic messaging with School Staff Outreach, consequences-framed basic 


20 Parents in all participating schools had the opportunity to opt out of the study through information provided via text 
messages and paper forms in early and mid-September 2017. Most opt-outs occurred prior to random assignment. These opt- 
outs were not considered to be attrition. The opt-outs that came after random assignment were considered to be attrition. 
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messaging with Goal Commitment, and no messaging (control). Based on the U.S. Department of Education, 
Institute of Education Sciences’ What Works Clearinghouse Standards Version 4.0, the overall and differential 
attrition rates correspond to tolerable levels of potential bias, even under cautious assumptions. Therefore, 
potential bias due to attrition seems unlikely to be driving the study results. 


Exhibit B.9. Attrition rates by study pathway 


(Oey Certaceyel Attrition Rate 


Benefits-framed basic messaging only throughout the year 14.1 
Consequences-framed basic messaging only throughout the year 14.7 
Benefits-framed basic messaging with School Staff Outreach 12.1 
Benefits-framed basic messaging with Goal Commitment messaging 15.8 
Consequences-framed basic messaging with School Staff Outreach 13.4 
Consequences-framed basic messaging with Goal Commitment messaging 13.5 
No-messaging control 11.7 
Overall 13.8 


SOURCE: Study records. 


The characteristics of students in the analysis sample were similar to those of the full sample of students at the 
time of the first random assignment in September 2017, suggesting that attrition did not affect the generalizability 
of the results (Exhibit B.10). 


Exhibit B.10. Background characteristics of students in the analytic sample compared to all students at 
baseline 


Analytic All students | Estimated 


Characteristics sample at baseline | difference 
Grade 2.5 2.5 0.0 0.382 
Female (percent) 48.2 48.3 0.1 0.778 
Race/ethnicity (percent) 
White, non-Hispanic 16.6 16.6 0.1 0.740 
Black, non-Hispanic 44.4 44.3 -0.1 0.663 
Asian, non-Hispanic 2.1 2.1 0.0 0.815 
Hispanic 32.1 32.3 0.2 0.559 
American Indian/Alaskan Native, Native Hawaiian, non- 0.6 0.6 0.0 0.949 
Hispanic 
Multi-racial, non-Hispanic 4.2 41 -0.1 0.485 
English language learner (percent) 14.1 14.2 0.1 0.788 
Special education status (percent) 15.9 15.9 0.1 0.813 
2016-17 Mathematics achievement (standardized) -0.231 -0.219 0.012 0.414 
2016-17 Reading/English/language arts achievement -0.221 -0.213 0.009 0.564 
(standardized) 
Chronically absent in 2016-17 (percent) 20.2 20.0 -0.2 0.533 
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Analytic All students | Estimated 
Characteristics sample at baseline | difference p-value 


Chronically absent from beginning of the year to random 14.4 14.1 -0.4 0.116 
assignment (percent) 


NOTES: Sample size = 108 schools; 7,069—23,133 students in the analytic sample; 7,684—25,834 students in the all students at baseline 
sample, with the sample size varying across different characteristics due to data availability. The analytic sample includes students who were 
enrolled for 100 percent of the school days and had complete demographic data. Achievement data were available only for students in 
grades 3-5, so sample sizes were smaller for achievement-related characteristics. The number of students for whom demographic data were 
available also varied. 

The analyses are based on a regression model controlling for block fixed effects. 


Exhibit B.11 shows the number of students in the sample throughout the study and sample attrition over the 
school year for the initial study conditions. Exhibit B.12 shows the same information for the intensified 
messaging groups. 


Exhibit B.11. Number of students in each initial study condition at the time of the first random 
assignment, removed from the analytic sample due to opting out, moving out of district, or not having 
complete demographic information, and ultimately included in the analytic sample, by district and 
overall 


Sample characteristic All districts 
Eligible students 7,100 8,746 4,920 8,866 29,632 
Students opted out before September randomization 430 1,015 284 1,060 2,789 
Students at random assignment 6,670 7,731 4,636 7,806 26,843 


Benefits-framed basic messaging 
Assigned to benefits-framed messaging 2,221 2,582 1,550 2,607 8,960 


Opted out after random assignment, or no longer in the 287 465 127 416 1,295 
district by the end of the school year, or did not have 
complete demographic information” 


Enrolled 100 percent of the school year with complete 1,934 2,117 1,423 2,191 7,665 
demographic information (analytic sample) 


Consequences-framed basic messaging 


Assigned to consequences-framed messaging 2,226 2,576 1,549 2,600 8,951 


Opted out after random assignment, or no longer in the 263 519 130 433 1,345 
district by the end of the school year, or did not have 
complete demographic information 


Enrolled 100 percent of the school year with complete 1,963 2,057 1,419 2,167 7,606 
demographic information (analytic sample) 


Control 
Assigned to control group 2,223 2,573 1,537 2,599 8,932 


2! Attrition categories are combined for privacy reasons. Most of the students in this group moved out of the district; opting 
out after random assignment and not having complete demographic information were relatively rare. 
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Sample characteristic All districts 


Opted out after random assignment, or no longer in the 237 334 107 393 1,070 
district by the end of the school year, or did not have 
complete demographic information 


Enrolled 100 percent of the school year with complete 1,986 2,239 1,431 2,206 7,862 
demographic information (analytic sample) 


Total sample 
Initial random assignment 6,670 7,731 4,636 7,806 26,843 
Opted out after random assignment, or no longer in the 787 1,318 363 1,242 3,710 


district by the end of the school year, or did not have 
complete demographic information 


Enrolled 100 percent of the school year with complete 5,883 6,413 4,273 6,564 23,133 
demographic information (analytic sample) 


SOURCE: Study and district administrative records. 


37 


Exhibit B.12. Number of students in each of the spring 2018 study conditions at the time of the second 
random assignment, removed from the analytic sample due to opting out, moving out of district, or not 
having complete demographic information, and ultimately included in the analytic sample, by district 
and overall 


Sample characteristic All districts 


Students from families below absence threshold on 
January 2 


Continuing in benefits-framed messaging 


In benefits-framed messaging on January 2 1,640 1,692 1,055 1,755 6,142 


Opted out after January 2, or no longer in the district by 135 170 65 215 585 
the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 1,505 1,522 990 1,540 5,557 
demographic information (analytic sample) 


Continuing in consequences-framed messaging 


In consequences-framed messaging on January 2 1,647 1,664 1,061 1,764 6,136 


Opted out after January 2, or no longer in the district by 124 184 58 229 595 
the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 1,523 1,480 1,003 1,535 5,541 
demographic information (analytic sample) 


Students from families above absence threshold on 
January 2 


Benefits-framed messaging with School Staff Outreach 


Assigned on January 2 253 336 233 410 1,232 


Opted out after January 2, or no longer in the district by 43 34 14 67 158 
the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 210 302 219 343 1,074 
demographic information (analytic sample) 


Consequences-framed messaging with School Staff 


Outreach 
Assigned on January 2 252 332 227 394 1,205 
Opted out after January 2, or no longer in the district by 29 33 20 81 163 


the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 223 299 207 313 1,042 
demographic information (analytic sample) 
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Sample characteristic All districts 


Benefits-framed messaging with Goal Commitment 


Assigned on January 2 253 336 234 406 1,229 


Opted out after January 2, or no longer in the district by 34 43 20 98 195 
the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 219 293 214 308 1,034 
demographic information (analytic sample) 


Consequences-framed messaging with Goal 


Commitment 
Assigned on January 2 246 330 222 394 1,192 
Opted out after January 2, or no longer in the district by 29 52 13 75 169 


the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 217 278 209 319 1,023 
demographic information (analytic sample) 


Control 
In Control on January 2 2,176 2,512 1,515 2,582 8,785 
Opted out after January 2, or no longer in the district by 190 273 84 376 923 


the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 1,986 2,239 1,431 2,206 7,862 
demographic information (analytic sample) 

Total Sample 
Students in sample on January 2 6,467 7,202 4,547 7,705 25,921 
Opted out after January 2, or no longer in the district by 584 789 274 1,141 2,788 


the end of the school year, or did not have complete 
demographic information 


Enrolled 100 percent of the school year with complete 5,883 6,413 4,273 6,564 23,133 
demographic information (analytic sample) 


SOURCE: Study and district administrative records. 


Baseline Equivalence 


It is important to demonstrate that the groups of students in each of the study conditions have similar 
characteristics at baseline. If they do not, for example, if one group is more “at risk” than another, then the 
estimates of the strategy’s effect may be biased. Random assignment assures that groups are statistically 
equivalent at baseline on average, but not necessarily in every instance, which is why it is good practice to always 
check for baseline equivalence. 


The study observed baseline equivalence on all measured background characteristics among initial basic 
messaging conditions, according to What Works Clearinghouse (WWC) standards. Exhibit B.13 shows the 
characteristics of students by initial messaging condition, in original units, and Exhibit B.14 shows the differences 
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in characteristics by initial messaging condition translated to standard deviation units.” Although there were 
statistically significant differences by basic messaging condition in the percentage of students who were White, 
Black, Hispanic, and who were English Language learners (Exhibit B.13), the magnitudes of these differences 
were all less than 0.05 standard deviations (Exhibit B.14).” 


Exhibit B.15 shows the background characteristics of students by intensified messaging group, in both original 
units and standard deviation units. In the analytic sample of students from families assigned to intensified 
intervention conditions, the study observed baseline equivalence on all background characteristics (Exhibit 
B.15). 


Although the WWC standards do not require it, the study’s analyses included covariate adjustments for all of the 
baseline characteristics. These adjustments help eliminate potential small biases due to imperfect 
randomization, and also improve the precision of the impact estimates. 


Exhibit B.13. Student background characteristics, by initial study condition 


Estimated difference 


Benefits- Benefits-framed 
Benefits- | Consequences- framed Consequences- basic vs. 
framed framed Control | basic vs. | framed basic vs. | Consequences- 
Characteristic basic mean basic mean mean control control framed basic 
Grade 2.5 2.5 2.5 0.0 0.0 0.0 
Female (percent) 48.9 48.9 47.7 1.2 1.1 0.0 
Race/ethnicity (percent) 
White, non-Hispanic 17.3 16.8 18.3 -1.0* -1.4* 0.5 
Black, non-Hispanic 42.2 44.5 43.5 -1.3* 1.0 -2.3* 
Asian, non-Hispanic 2.1 2.0 2.1 0.1 -0.1 0.1 
Hispanic 33.6 32.0 31.7 1.9* 0.2 1.6* 
American 0.5 0.6 0.5 0.0 0.1 -0.1 
Indian/Alaskan 


Native, Native 

Hawaiian, non- 

Hispanic 

Multi-racial, non- 4.2 4.0 3.9 0.4 0.2 0.2 
Hispanic 


English language learner 14.8 14.4 13.5 1.3* 0.9 0.4 
(percent) 

Special education status 15.3 15.8 16.3 -1.0 -0.5 -0.5 
(percent) 

2016-17 Mathematics -0.178 -0.202 -0.181 0.003 -0.021 0.024 
achievement 

(standardized) 


22 WWC standards for baseline equivalence are based on differences expressed in standard deviation units. 

23 According to WWC standards, a study demonstrates baseline equivalence if differences in baseline characteristics are no 
greater than 0.05 standard deviations (WWC Standards Handbook, version 4.1, p. 13). Differences between 0.05 and 0.25 
standard deviations are also acceptable but require statistical adjustment (e.g., inclusion as covariates in outcome analyses). 


40 


Estimated difference 


Benefits- Benefits-framed 
Benefits- | Consequences- framed Consequences- basic vs. 
framed framed basic vs. | framed basic vs. | Consequences- 
(Wiktevacoucie basic mean basic mean control control framed basic 
2016-17 Reading/ -0.186 -0.215 -0.165 -0.021 -0.049 0.029 
English/language arts 
achievement 
(standardized) 
Chronically absent in 19.0 18.6 18.3 0.6 0.2 0.4 
2016-17 (percent) 
Chronically absent from 13.0 13.1 13.0 0.0 0.0 0.0 
beginning of the year to 
random assignment 
(percent) 


SOURCE: District administrative records and SchoolMessenger archival data. 

NOTES: Sample size = 108 schools; 2,359—7,699 students in benefits-framed basic messaging; 2,296—7,629 students in consequences-framed 
basic messaging; 2,428—7,884 students in Control. Achievement data were available only for students in grades 3-5, so sample sizes were 
smaller for achievement-related characteristics. 

The analyses are based on a regression model controlling for block fixed effects. Control group means are unadjusted means; messaging 
group means were computed by adding the estimated differences to the control group means. 

* Indicates that the average student characteristic for the messaging group (benefits-framed basic messaging or consequences-framed basic 
messaging) is statistically significantly different from the average student characteristic for the no-messaging control group, or that the 
difference in average student characteristic for the two basic messaging groups (benefits-framed or consequences-framed) is statistically 
significant (p < .05). 


Al 


Exhibit B.14. Differences in student background characteristics, by initial study condition 


Estimated difference in standard deviation units 


Benefits-framed Consequences-framed Benefits- basic vs. 


(Wiktevacaucule basic vs. control basic vs. control Consequences- basic 
Grade -0.010 -0.010 0.000 
Female (percent) 0.023 0.023 0.001 
Race/ethnicity (percent) 
White, non-Hispanic -0.025* -0.037* 0.012 
Black, non-Hispanic -0.026* 0.020 -0.046* 
Asian, non-Hispanic 0.004 -0.005 0.009 
Hispanic 0.040* 0.005 0.035* 
American Indian/Alaskan Native, Native -0.006 0.012 -0.018 
Hawaiian, non-Hispanic 
Multi-racial, non-Hispanic 0.019 0.009 0.010 
English language learner (percent) 0.037* 0.025 0.012 
Special education status (percent) -0.027 -0.012 -0.015 
2016-17 Mathematics achievement 0.003 -0.022 0.025 
(standardized) 
2016-17 Reading/English/language Arts -0.022 -0.052 0.030 
achievement (standardized) 
Chronically absent in 2016-17(percent) 0.016 0.006 0.010 
Chronically absent from beginning of the year 0.000 0.001 -0.001 


to random assignment (percent) 


SOURCE: District administrative records and SchoolMessenger archival data. 

NOTES: Sample size = 108 schools; 2,359—7,699 students in benefits-framed basic messaging; 2,296—7,629 students in consequences-framed 
basic messaging; 2,428—7,884 students in Control. Achievement data were available only for students in grades 3-5, so sample sizes were 
smaller for achievement-related characteristics. 

The analyses are based on a regression model controlling for block fixed effects. 

* Indicates that the average student characteristic for the messaging group (benefits-framed basic messaging or consequences-framed basic 
messaging) is statistically significantly different from the average student characteristic for the control group, or that the difference in average 
student characteristic for the two basic messaging conditions (benefits-framed or consequences-framed) is statistically significant (p < .05). 
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Exhibit B.15. Student background characteristics, by intensified messaging group 


Estimated difference 


School Staff 
School Staff Goal Outreach vs. Std. dev. units 
Outreach | Commitment efor | Outreach vs. 

Characteristic mean mean Commitment Commitment 
Grade 2.4 2.4 0.0 0.000 
Female (percent) 46.9 48.7 -1.7 -0.034 
Race/ethnicity (percent) 

White, non-Hispanic 15.7 15.9 -0.2 -0.006 

Black, non-Hispanic 49.1 50.1 1.0 -0.020 

Asian, non-Hispanic 1.1 1.3 -0.1 -0.012 

Hispanic 29.1 27.5 1.6 0.036 

Multi-racial, non-Hispanic 4.4 4.8 -0.3 -0.015 
English language learner (percent) 12.5 11.7 0.8 0.024 
Special education status (percent) 19.6 19.4 0.2 0.004 
2016-17 Mathematics achievement (standardized) -0.464 -0.426 -0.038 -0.039 
2016-17 Reading/English/language Arts -0.415 -0.457 0.042 0.043 
achievement (standardized) 
Chronically absent in 2016-17 (percent) 43.2 45.1 -1.8 -0.037 
Chronically absent from beginning of the year to 23.8 25.8 -2.0 -0.045 


random assignment (percent) 


SOURCE: District administrative records and SchoolMessenger archival data. 

NOTES: Sample size = 108 schools; 570—2,116 students in School Staff Outreach; 572—2,057 students in Goal Commitment messaging. 
Achievement data were available only for students in grades 3-5, so sample sizes were smaller for achievement-related characteristics. 
The analyses are based on a regression model controlling for block fixed effects. Goal Commitment group means are unadjusted means; 
School Staff Outreach group means were computed by adding the estimated differences to the Goal Commitment group means. 

No groups were statistically significantly different from each other at p < .05. 
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Data Collection and Measures Used in the Study 


Exhibit B.16 displays the study’s data sources. These data sources allowed the study team to construct measures 
of outcomes, school and student background characteristics, implementation of the texting strategy, and cost. 


Exhibit B.16. Data sources 


Type of data 


Attendance data 


Source 


District student information system 


Uses in the study and analysis 


Calculate attendance rates from October 
1 through end of fall semester to identify 
students above and below the 8 percent 
threshold for randomizing families to 
intensified messaging. Construct two 
attendance outcome measures for 
impact analyses: (1) a binary measure of 
chronic absence (absent 10 percent or 
more of instructional days), and (2) a 
continuous measure of number of days 
absent. Both outcome measures (the 
primary chronic absence and the 
secondary days absent) were calculated 
over three periods for different impact 
analyses: (1) October 1 to end of fall 
semester for the Fall 2017 outcomes; (2) 
January 2 to end of spring semester for 
the Spring 2018 outcomes; and (3) 
October 1 to the end of spring semester 
for the school year 2017-18 outcomes. 


Academic achievement 


District student information system 


Create achievement outcomes for 
impact analyses (school year 2017-18): 
standardized (across district and grade) 
reading and mathematics scores. 


Student rosters and background 
characteristics, including prior-year 
attendance and academic achievement; 
free or reduced-price lunch program 
eligibility or economically 
disadvantaged indicator, race/ethnicity, 
eligibility for special education services 


District student information system 


Describe the sample in terms of 
background characteristics (Fall 2017); 
assess generalizability; stratify for 
randomization; determine pre- 
randomization subgroups used for 
exploratory analyses; create covariates 
for impact analyses. 


School characteristics data 


Common Core of Data, Office for Civil 
Rights data, district student information 
system 


Describe the sample. 


Attendance and text messaging system 
data 


Text messaging provider system 


Monitor intervention delivery which 
was based on daily attendance data 
from districts; gauge intervention 
fidelity, uptake, and dosage. 
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Type of data Source Uses in the study and analysis 


School Attendance Counselor Log Study team-created protocol completed Describe the role of the attendance 

by school staff, once during school counselor and the school staff outreach 

attendance counselor training and three activities (part of implementation 

times in Spring 2018 analysis); describe the time spent in 
parent outreach (part of cost analysis); 
and describe business as usual 
attendance practices used by 
participating schools (service contrast). 


Text messaging provider cost Study team accounting system Estimate cost of delivering the text 

information messages by the text messaging 
provider. 

Study team labor cost information Study team accounting system Estimate cost of supporting the delivery 


of the text messages by the study team. 


Information about districts’ cost to Cost interviews; district IT department Estimate cost of setting up and carrying 

implement text messaging out daily data transfers to text 
messaging provider by district IT 
departments. 


SOURCE: Authors’ compilation. 


Outcome Measures 


Chronic absence was the primary outcome for the study because it is the absence measure that is most policy 
relevant, as most states are using it for their accountability systems under The Every Student Succeeds Act.”4 It 
was defined as missing 10 percent or more instructional days.*° A secondary attendance measure was the 
number of days absent, used to confirm findings were similar when looking at average number of absences. Both 
measures have been used in previous studies testing the impact of informational interventions on student 
attendance.” The study created these measures using district attendance data for the fall (October 1 through the 
end of fall semester), for the spring (January to the end of spring semester), and for the full implementation 
period (October 1 through the end of spring semester). Multiple comparisons adjustments are not needed in this 
case because there is only one primary outcome measure (chronic absence) within the same domain (attendance). 


Academic achievement outcomes were more exploratory because they were only available for part of the study 
sample. Test scores were available for grades 3 to 5 for English/language arts and mathematics in all districts. Because 
the tests used by districts differed by state, the study team standardized the student test scores by using each district’s 
mean and standard deviation for each grade level. Due to family-level randomization within schools, the students in 
different messaging groups are, by design, compared only to other students in the same schools (and thus the same 
district and state). Therefore, students were compared to others who took the same achievement test. 


24 Bauer, Liu, Schanzenbach, & Shambaugh, 2018. 

25 Chronic absence is commonly defined as missing 10 percent or more of a school year by states and school districts and 
organizations working on school attendance (see, for example, https://www.attendanceworks.org/chronic-absence/the- 
problem/). When the U.S. Department of Education instituted reporting of chronic absenteeism in the 2013-2014 Office for 
Civil Rights Data Collection, however, the measure was defined as the proportion of students who were absent 15 or more 
days of the school year. All definitions of chronic absence include both unexcused and excused absences, due to the 
assumption that less time in school affects students no matter the reason. 

26 Balu, Porter, & Gunton, 2016; Rogers & Feller, 2018. 
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Background Characteristics, Implementation Measures, and Cost 


The study team collected school and student background characteristics to describe the sample. The analyses 
also used the student background characteristics as covariates in the impact analyses. 


The data from the text messaging provider included daily information about the text messages sent and 
delivered. The study team used these data to construct measures of fidelity and dosage of the automated text 
messaging components to help understand how well the messaging was implemented. Fidelity was defined as 
the rate at which intended messages were sent to the correct families on a daily basis, across the fall and spring 
of the study year. 


Exhibit B.17 shows how the study determined whether intended messages were sent to at least one correct 
parent. Dosage was defined as the rate at which intended messages were delivered to at least one correct parent. 
Delivered messages were those recorded by the system as having been sent and successfully delivered to a local 
mobile phone carrier. 
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Exhibit B.17. Definition of how rate was calculated for intended messages sent to correct families 


Message Eligible for message Description of the possible messaging scenarios Correct | Incorrect 
Informational Everyone assigned to benefits- or Message sent to at least one parent of an eligible student v 
messages and tips consequences-framed basic 
(Sundays) messaging at baseline 
In district dataset on day of message Message not sent to at least one parent (for example, because SchoolMessenger v 
delivery system failed or message wasn't scheduled correctly) 


At least one parent has not requested 
that the text messages be stopped 


Same-day Same as above Student is absent that day, the absence is not more than third consecutive 
notifications 1-3 days absence, and same-day notification message sent to at least one parent 


(daily on school days) Student is not absent that day, and same-day notification message not sent 


Student is absent that day, student has been absent 4 or more consecutive 
days, and same-day notification message not sent 


Student is absent that day, the absence is not more than third consecutive 
absence, and same-day notification message not sent 


Student is absent that day, the student has been absent for 4 or more v 
consecutive days, and same-day notification message sent 


Student is present that day and a same-day notification message sent 


Welcome back (daily Same as above? Student was absent yesterday, present today, and message sent 


on school days) 
Student was absent yesterday, absent today, and message not sent v 


Student was not absent yesterday, and message not sent v 


Student was absent yesterday, absent today, and message sent v 


Student was absent yesterday, present today, and message not sent 
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Message Eligible for message 


4th day notifications Same as above 
(daily on school days) 


Description of the possible messaging scenarios 


Student is absent today, today’s absence is the fourth consecutive, and 
message sent 


Correct 
v 


Incorrect 


Student is absent today, today’s absence is not the fourth consecutive, and 
message not sent 


Student is present today and message not sent 


Student is absent today and today’s absence is the fourth consecutive, and 
message not sent 


Student is absent today and today’s absence is not the fourth consecutive, and 
message sent 


Goal Commitment Same as above, plus randomly 
message (Sundays) assigned to Goal Commitment 
messaging 


Message sent to at least one parent of an eligible student 


Message not sent to at least one parent of an eligible student (for example, 
because SchoolMessenger system failed, message wasn't scheduled correctly) 


Goal feedback Same as above, plus randomly 
(Fridays) assigned to Goal Commitment 
messaging 


Student had perfect attendance (indicator shows no absences that week), and 
congratulations/perfect attendance message sent 


Student had imperfect attendance (indicator shows 1+ absences that week), and 
“try harder” message sent 


Student had perfect attendance (indicator shows no absences that week), and 
“try harder” message sent 


Student had imperfect attendance (indicator shows 1+ absences that week), and 
“congratulations” message sent 


Parents indicated they did not want to commit to perfect attendance and 
congratulations/perfect attendance message sent 


Parents indicated they did not want to commit to perfect attendance and “try 
harder” message sent 


Message not sent to at least one parent of an eligible student (for example, 
because SchoolMessenger system failed) 


School Staff Outreach Same as above, plus randomly 
(Spring) assigned to School Staff Outreach 


At least one message sent by school staff 


No messages sent by school staff 
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The school attendance counselor log provided data about the implementation of School Staff Outreach as well as 
measures of service contrast (that is, typical attendance outreach-related practices in study schools). 


The staff who conducted School Staff Outreach completed school attendance counselor logs in early February, 
early March, and early April. The logs asked the attendance counselor about outreach activities to a sample of 
families across all study conditions during the prior month. Exhibit B.18 shows the response rate for each of the 
three logs, and overall. 


Exhibit B.18. School staff outreach attendance counselor log response rates 


Log 1 1 Ku) Log 3 Overall 
Number of schools 108 108 108 108 
Number of attendance counselor logs 118 118 118 354 
Response rate (percent) 93.2 94.1 95.8 94.4 


SOURCE: Authors’ compilation. 


Measures of the costs of the texting strategy were created from interview protocols developed by the study team 
and administered to district IT staff. The study team also captured cost data from AIR’s accounting system, which 
included costs for study team staff for supporting implementation (such as conducting School Staff Outreach 
trainings) and the text messaging provider. 


Minimum Detectable Effects 


Minimum detectable effects are the smallest differences between one texting group and another (or the no- 
messaging control group) that the study has a good chance of finding to be statistically significant. The study 
team used actual sample sizes and data collected to estimate the minimum detectable effects for the comparisons 
of the four adaptive interventions (that is, the four versions of the adaptive texting strategy), as well as the 
comparisons between the two basic messaging approaches and the two intensified messaging approaches. The 
minimum detectable effects for these comparisons were calculated for the two main analysis samples for the 
study - the overall student sample and the subsample of students in families with a history of high absences 
(hereafter referred to as “students with a history of high absences”). 


The minimum detectable effects are shown in Exhibits B.19 to B.24. The size of the differences in attendance 
outcomes that this study could detect with the total analytic sample was similar to those found in prior studies 
that tested informational attendance-related strategies.”” The detectable effects are relatively small by 
conventional standards in education (less than 0.10 standard deviations), but they are still policy relevant 
magnitudes, especially given that the strategy tested in this study is expected to be relatively low cost. 


27 For example, the effect on total number of days absent in Rogers, Duncan, Wolford, Ternovski, Subramanyam, & Reitano 
(2017) was 0.03. 
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Exhibit B.19. Minimum detectable effects for comparing the four versions of the adaptive texting 
strategy to the no-messaging control condition and to each other on attendance outcomes for the 
overall sample 


Minimum detectable effect 


Comparison Chronic absence Days absent 
Benefits-framed basic with School Staff Outreach (AI-1) vs. control 0.05 0.04 
Benefits-framed basic with Goal Commitment messaging (AI-2) vs. control 0.05 0.04 
Consequences-framed basic with School Staff Outreach (AI-3) vs. control 0.05 0.04 
Consequences-framed basic with Goal Commitment messaging (AI-4) vs. control 0.05 0.04 
AI-1 vs. AI-2 0.04 0.05 
AI-1 vs. AI-3 0.04 0.05 
AI-1 vs. AI-4 0.04 0.05 
AI-2 vs. AI-3 0.03 0.05 
AI-2 vs. AI-4 0.04 0.05 
AI-3 vs. AI-4 0.03 0.05 


SOURCE: Authors’ compilation. 
NOTES: The minimum detectable effect reported for chronic absence is Cohen’s h. The minimum detectable effect for number of days absent 
is Cohen’s d. 


Exhibit B.20. Minimum detectable effects for comparing the four versions of the adaptive texting 
strategy to the no-messaging control condition and to each other on attendance outcomes for students 
with a history of high absences 


Minimum detectable effect 


Comparison Chronic absence Days absent 
Benefits-framed basic with School Staff Outreach (AI-1) vs. control 0.09 0.08 
Benefits-framed basic with Goal Commitment messaging (AI-2) vs. control 0.09 0.08 
Consequences-framed basic with School Staff Outreach (AI-3) vs. control 0.09 0.08 
Consequences-framed basic with Goal Commitment messaging (AI-4) vs. control 0.09 0.08 
AI-1 vs. AI-2 0.07 0.10 
AI-1 vs. AI-3 0.05 0.10 
AI-1 vs. AI-4 0.09 0.10 
AI-2 vs. AI-3 0.08 0.10 
AI-2 vs. AI-4 0.08 0.10 
AI-3 vs. AI-4 0.09 0.10 


SOURCE: Authors’ compilation. 
NOTES: The minimum detectable effect reported for chronic absence is Cohen’s h. The minimum detectable effect for number of days absent 
is Cohen’s d. 
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Exhibit B.21. Minimum detectable effects for comparing the four versions of the adaptive texting 
strategy to the no-messaging control condition and to each other on achievement outcomes for the 
overall sample 


Minimum detectable effect 


Reading Math 
Comparison achievement achievement 
Benefits-framed basic with School Staff Outreach (AI-1) vs. control 0.07 0.07 
Benefits-framed basic with Goal Commitment messaging (AI-2) vs. control 0.07 0.07 
Consequences-framed basic with School Staff Outreach (AI-3) vs. control 0.07 0.07 
Consequences-framed basic with Goal Commitment messaging (AI-4) vs. control 0.07 0.07 
AT-1 vs. AI-2 0.08 0.08 
AI-1 vs. AI-3 0.08 0.08 
AI-1 vs. AI-4 0.08 0.08 
AI-2 vs. AI-3 0.08 0.08 
AI-2 vs. AI-4 0.08 0.08 
AI-3 vs. AI-4 0.08 0.08 


SOURCE: Authors’ compilation. 
NOTES: The minimum detectable effect for reading and mathematics achievement is Cohen’s d. 


Exhibit B.22. Minimum detectable effects for comparing the four versions of the adaptive texting 
strategy to the no- messaging control condition and to each other on achievement outcomes for 
students with a history of high absences 


Minimum detectable effect 


Reading Math 
Comparison achievement achievement 
Benefits-framed basic with School Staff Outreach (AI-1) vs. control 0.15 0.15 
Benefits-framed basic with Goal Commitment messaging (AI-2) vs. control 0.15 0.15 
Consequences-framed basic with School Staff Outreach (AI-3) vs. control 0.14 0.14 
Consequences-framed basic with Goal Commitment messaging (AI-4) vs. control 0.14 0.14 
AI-1 vs. AI-2 0.17 0.17 
AI-1 vs. AI-3 0.16 0.16 
AI-1 vs. AI-4 0.16 0.16 
AI-2 vs. AI-3 0.16 0.16 
AI-2 vs. AI-4 0.16 0.16 
AI-3 vs. AI-4 0.16 0.16 


SOURCE: Authors’ compilation. 
NOTES: The minimum detectable effect for reading and mathematics achievement is Cohen’s d. 
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Exhibit B.23. Minimum detectable effects for comparing the basic messaging approaches to the no- 
messaging control condition and to each other on attendance outcomes for the overall sample and for 
students with a history of high absences 


Minimum detectable effect 


Students with a history of high 
Overall sample absences 


Comparison Chronic absence | Days absent Chronic absence Days absent 
Any basic messaging (benefits- or 0.04 0.04 0.07 0.07 
consequences-framed) vs. control 

Benefits-framed basic messaging vs. control 0.04 0.04 0.08 0.08 
Consequences-framed basic messaging vs. 0.04 0.04 0.08 0.08 
control 

Benefits-framed basic messaging vs. 0.04 0.04 0.08 0.08 


consequences-framed basic messaging 


SOURCE: Authors’ compilation. 
NOTES: The minimum detectable effect reported for chronic absence is Cohen’s h. The minimum detectable effect for number of days absent 
is Cohen’s d. 


Exhibit B.24. Minimum detectable effects for comparing the intensified messaging approaches to each 
other on attendance outcomes for the overall sample and for students with a history of high absences 


Minimum detectable effect 


Students with a history of high 
Overall sample absences 


Comparison Chronic absence Days absent | Chronic absence | Days absent 


School Staff Outreach vs. Goal Commitment 0.02 0.02 0.06 0.06 


SOURCE: Authors’ compilation. 
NOTES: The minimum detectable effect reported for chronic absence is Cohen’s h. The minimum detectable effect for number of days absent 
is Cohen’s d. 
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Impact Analysis Approaches 


This study was designed to estimate the impact of the different versions of the adaptive messaging strategy on 
student attendance at two points in the year (mid-year and end of year) and achievement at the end of the year. 
Exhibit B.25 provides a high-level summary of the types of analyses reported in Appendix C and that provided the 
basis for the findings summarized in the main body of the report. Appendix C also includes results for the 
secondary attendance outcome, number of days absent. Additional exploratory and sensitivity analyses and results 
are described in Appendix E. 


All impact analyses were conducted with the full student analytic sample defined above in “Random Assignment 
and Analytic Sample,” and not just those for whom the sending and receipt of the messages was confirmed. This 
means the study estimated the impact of parents being offered the text messaging (the “intent to treat”) rather 
than the impact of using the messages, because the former is a more realistic way in which districts would 
consider adding a strategy like this and more accurately reflects the cost of the strategy. 


The impact models included a number of covariates to take into account the way random assignment was 
conducted and the possibility of differences in student characteristics across the messaging versions by chance. 
The student-level characteristics included free or reduced-price lunch program eligibility, student ethnicity/race, 
special education status, and prior year academic achievement (only for achievement outcome analysis). Other 
covariates represented the blocking variables used in the randomization: having a history of high absences, 
having at least one parent with a working mobile phone with short message service (SMS) at baseline, and school 
indicator variables. (Having a history of high absences was omitted as a covariate for analyses with the sub- 
sample of students with a history of high absences). Using the randomization block as a covariate makes the 
analysis a “fixed effects” model, ensuring that only students with the same observable characteristics are 
compared to each other (for example, from the same school, same special education status, etc.). 


The analyses correct for multiple comparisons because they compared different text messaging strategies to each 
other.” The analysis corrected for four comparisons for basic messaging (any of the two basic messaging 
strategies compared to control, each of the two basic messaging strategies separately compared to control, and 
the two basic messaging strategies compared to each other). The analysis of the adaptive interventions compared 
to the no-messaging control condition also corrected for four comparisons (each adaptive intervention compared 
to control), while the analysis comparing the adaptive interventions to each other corrected for six comparisons. 
The analyses did not include multiple comparison corrections for the number of outcomes. The WWC Handbook 
specifies that such adjustments are only needed for multiple primary outcomes within the same domain. Chronic 
absence and number of days absent are within the same outcome domain (attendance), but only chronic 
absence was considered a primary outcome in this study. Reading and mathematics achievement are each 
considered to be in separate domains. 


8 The study’s approach to correcting for multiple comparisons was guided by the What Works Clearinghouse (WWC) 
Procedures Handbook (version 4.1, Appendix F, pp. F-3 to F-7). However, because the WWC has not yet addressed the issue 
of multiple comparisons for SMART designs specifically, the study adopted a conservative approach. 
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Exhibit B.25. Summary of impact analyses reported in the main body of the report and Appendix C 


Research question Outcomes Comparisons of groups WY Cdavoyel 
1.a Do the four combinations of | Primary attendance Compare each adaptive Regression analysis (students 
basic and intensified messaging outcome: chronic absence intervention to the no- within blocks, blocks as fixed 
(i.e., the four adaptive (missed 10 percent or messaging control group. effects) with inverse 


interventions) improve end-of- —_ more of school days) 
year attendance or achievement between October 1 and 


probability weighting to 
account for sequential 


Results reported for the 
overall sample and 


when compared to no- end of year). subsample of students with a randomization. 
messaging? Secondary attendance history of high absences. 
outcome: number of days 
absent between October 1 
and end of year). 
1.b Do the two basic messaging Chronic absence Benefits-framed basic or Regression analysis (students 
approaches (benefits-framed and (primary), and number of consequences-framed basic _ within blocks, blocks as fixed 
consequences-framed days absent (secondary) vs. no-messaging control effects); Wald tests to compare 
messaging) improve end-of-fall between October andend », aehietamied basieve parameters based on the 
attendance compared to no of December 2017. Control sample estimate” 
messaging? 
Consequences-framed basic 
vs. no-messaging control 
Results reported for the 
overall sample and 
subsample of students with a 
history of high absences. 
1.c Did adding intensified Chronic absence between Any basic messaging Regression discontinuity 
messaging to basic messaging in January 2 andend of year. (benefits- or consequences- _ analysis comparing the 
the spring reduce absence more framed) vs. any intensified chronic absence outcome for 
than basic messaging would messaging (School Staff students from families just 
have alone, for students with Outreach or Goal below the 8 percent threshold 
higher absences in the fall? Commitment) to qualify for intensive 


messaging to those just above 
the 8 percent threshold to 
qualify for intensive 


Results reported for the 
overall sample and 
subsample of students with a 


history of high absences. messaging. 


2° The Wald test is used in statistical inference and hypothesis testing. It is a statistical test of estimated parameters in a 
model, with the null hypothesis that a set of parameters is equal to some designated value. The Wald test can be used to test 
multiple hypotheses about multiple parameters simultaneously (https://www.air.org/sites/default/files/EdSurvey- 


WaldTest.pdf). 
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Research question 


(Ob ikea) nite) 


Comparisons of groups 


2.a Do the four combinations of Primary: chronic absence Compare adaptive 


basic and intensified messaging 
(i.e., the four adaptive 
interventions; Als) have different 
effects on end-of-year 
attendance when compared to 
each other? 


(missed 10 percent or 
more of school days) 
between October 1 and 
end of year 


Secondary: number of 
days absent between 
October 1 and end of year 


interventions to each other. 
All vs. AI2 
All vs. AI3 
All vs. Al4 
AI2 vs. AI3 
AI2 vs. AI4 
AI3 vs. AI4 


Results reported for the 
overall sample and 
subsample of students with a 
history of high absences. 


AY Cordaterel 


Regression analysis (students 
within blocks, blocks as fixed 
effects) with inverse 
probability weighting to 
account for sequential 
randomization. 


2.b Do the two basic messaging 
approaches (benefits-and 
consequences-framed 
messaging) have different effects 
on end-of-fall attendance when 
compared to each other? 


Chronic absence 
(primary), and number of 
days absent (secondary) 
between October and 
December 2017. 


Benefits-framed basic 
messaging vs. consequences- 
framed basic messaging. 


Results reported for the 
overall sample and 
subsample of students with a 
history of high absences. 


Regression analysis (students 
within blocks, blocks as fixed 
effects); Wald tests to compare 
parameters based on the 
sample estimate. 


2.c For children above the 8 
percent absence threshold in the 
fall despite basic messaging, 
does providing school staff- 
initiated parent outreach and 
interpersonal support (School 
Staff Outreach) or goal 
commitment messaging with 
additional tips and resources 
(Goal Commitment messaging) 
affect attendance? 


Chronic absence (10 
percent or more school 
days missed) between 
January 2 and the end of 
spring semester 2018 
(grades K-5). Number of 
days absent as secondary 
attendance outcome. 


School Staff Outreach vs. 
Goal Commitment messaging 


Results reported for the 
overall sample, subsample of 
students with a history of 
high absences, and 
subsample of students 
whose home language was 
English. 


Regression analysis (students 
within blocks, blocks as fixed 
effects) with inverse 
probability weighting to 
account for sequential 
randomization. 


3. Do the four adaptive 
interventions have effects on 
academic achievement when 
compared to no-messaging 
control? 


Math and reading 
achievement (grades 3-5) 
at the end of spring 
semester 2018 


Compare each adaptive 
intervention to the no- 
messaging control group. 


Results reported for the 
overall sample and 
subsample of students with a 
history of high absences. 


Regression analysis (students 
within blocks, blocks as fixed 
effects) with inverse 
probability weighting to 
account for sequential 
randomization. 


SOURCE: Authors’ compilation. 
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Analytical Models for Comparing Adaptive Interventions to No-Messaging Control 
Condition and to Each Other 


The study used regression models to estimate the impact of the four adaptive interventions compared to the no- 
messaging control condition and to each other. A model for the number of days absent outcome is described 
here using a linear specification (a sensitivity analysis was also conducted using Poisson models, as described in 
Appendix E). The analytical model for the chronic absence outcome is similar but used a probit model. 


Let Y;; be the number of days absent outcome for student i in block j. Let AY? € {—1,0,1} denote the basic 
(1) 
ij 

—1if the student/family was assigned to consequences-framed basic 


messaging assigned in the first randomization such that A;;’ = 0 if the student/family was assigned to the no- 


messaging control condition, A\;) = 
{;) = 1 if the student/family was assigned to benefits-framed basic messaging. Let A‘) € {—1,0,1} 
denote the intensified messaging assigned at the second stage such that Ai, = —1 if the student/family was 
assigned to Goal Commitment messaging, AY? = 1 if the student/family was assigned to School Staff Outreach, 


messaging, and A 


and AY = 0 if the student/family was assigned to the no-messaging control condition in the first randomization. 


As a result, the following five messaging conditions were embedded in the sequential multiple assignment 
randomized trial (SMART): 


1, ©0n*rol which assigned students and their families to the no-messaging control condition in the first 
randomization and continued to the end of the study. 

2. 74! which assigned students and their families to benefits-framed basic messaging in the first 
randomization. Families with students above the 8 percent absence threshold were assigned to School Staff 
Outreach in the second randomization while families whose students were all below the absence threshold 
continued with the benefits-framed basic messaging. 

3. 14!? which assigned students and their families to benefits-framed basic messaging in the first 
randomization. Families with students above the 8 percent absence threshold were assigned to Goal 
Commitment in the second randomization while families whose students were all below the absence 
threshold continued with benefits-framed basic messaging. 

4. 74!3 which assigned students and their families to consequences-framed basic messaging in the first 
randomization. Families with students above the 8 percent absence threshold were assigned to School Staff 
Outreach in the second randomization while families whose students were all below the absence threshold 
continued with consequences-framed basic messaging. 

5. 14!4 which assigned students and their families to consequences-framed basic messaging in the first 
randomization. Families with students above the 8 percent absence threshold were assigned to Goal 
Commitment in the second randomization while families whose students were all below the absence 
threshold continued with consequences-framed basic messaging. 


First, consider all the students/families who were consistent with strategy 74!1. This strategy included (1) all 
students/families assigned to benefits-framed basic messaging in the first randomization who were below the 
absence threshold at the end of the fall semester and continued with benefits-framed basic messaging (with 
probability of 1.00) and (2) those students/families who were above the absence threshold at the end of the fall 
and were assigned to School Staff Outreach in the second randomization (with probability of 0.50). Since only 
half of the students/families who were above the absence threshold were assigned to School Staff Outreach (the 
other half were assigned to Goal Commitment messaging), students/families who were above the absence 
threshold would be under-represented when compared to those who were below the threshold. The analysis 
corrected for this under-representation by assigning a weight, W;;, to student i in block j. The weight W;; is the 
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inverse of the probability of being randomized to the basic and intensified messaging strategies that the 
student/family received. Because students/families were randomized to benefits-framed basic messaging, 
consequences-framed basic messaging, and the no-messaging control condition in equal proportions (with 
randomization probability 1/3 for each condition), a student/family who was below the absence threshold at the 
end of the fall semester and who therefore continued with their basic messaging strategy with probability of 1.00 
received a weight W;; = 3. A student/family who was above the absence threshold at the end of the fall semester 
and was then assigned to School Staff Outreach with randomization probability 0.50 received a weight of W;; = 
6. Students/families who were assigned to the no-messaging control condition and continued with the control 
condition received a weight of W;; = 3. 


Further adjustments were required to appropriately estimate the means of all the adaptive interventions 
simultaneously. Ifa student/family was assigned benefits-framed basic messaging in the first randomization and 
was below the absence threshold at the end of the fall semester, this student’s/family’s assignments would be 
consistent with both 74!" and 14”. To analytically correct for this, their observation is counted twice, once for 
both adaptive interventions. This was done by creating a new dataset that had two identical observations for 
each student/family that was below the absence threshold after the fall semester (i.e., students/families who 
would continue with their basic messaging assignment) and the original data for each student/family who was 
above the absence threshold. Then for the students/families that were below the absence threshold after the fall 
semester, we defined the indicator of the intensified messaging strategy AY? = 1 (i.e., School Staff Outreach) for 


one of the identical observations and AY? = —1 (i.e., Goal Commitment messaging) for the other. 
The model for the number of days absent outcome is then: 


1 2 1 2 1 
B.1. Yij = Bot BAG? + BoA? + BAP AD + EX +, + vi 


where X Ss is a grand mean centered vector of baseline student covariates, y; is a fixed effect for block j, and vj; is 
the within-block error for child i in block j that is taken to be normally distributed with mean O and variance o?. 
Then the expected number of days absent for each of the adaptive interventions can be written as a linear 
combination of the £’s such that 


HA" = Bot Bi + Bot Bs 
uA? = Bot Bi— Bo— Bs 
HAS = Bo — Bi t+ Bo - Bs 
walt = By — By — Ba + Bs 


Control _ 
H = Bo- 


The regression coefficients were estimated using weighted least squares by minimizing 
2 2 ah 
B.2. >) WY — Bo- BAD — BoA — BADAP — xP —y,) 


Standard errors were estimated using the robust sandwich standard errors, which account for the replication of 
records for students below the absence threshold. These estimates were used to test the null hypothesis that 
there is no difference in the mean outcome between each adaptive intervention and the control condition or 
between the different adaptive interventions. For example, the estimator for u4/1 — y4! = 2B, + 2B, can be 
shown to have an approximately normal sampling distribution, and a t-test statistic based on this result can be 


57 


used to test the null hypothesis that there is no difference in mean outcome between following 14” and 14/2. 


The study accounted for multiple comparisons with a Bonferroni correction, specifically, for four different tests 
to compare the adaptive intervention means to the control group mean, and six different tests when comparing 
the four adaptive intervention means to each other. 


Analytical Models for Comparing Basic Messaging Approaches to No-Messaging Control 
and Each Other 


The following model was used to estimate the effect on basic messaging (benefits- or consequences-framed) on 
chronic absence: 


B.3. Pr(¥j;=1) = (0) + G.MSGy + é7X +y;) 


where Yj; is an indicator of whether child i in block j was chronically absent ( Y;; equals 1 if the student was 
chronically absent and 0 if the student was not chronically absent), MSG;; is an indicator for whether 
child/family i in block j was assigned to either benefits- or consequences-framed basic messaging, X 7 is a vector 
of student characteristics, and y; is a fixed effect for block j. @() is defined as the cumulative distribution 
function of the standard normal distribution. 


A Wald test determined whether assignment to basic messaging (benefits- or consequences-framed) had a 
significant effect on chronic absence in the fall semester by testing whether ¢, is significantly different than zero. 


The model to estimate the separate effect of each of the two basic messaging approaches (versus control) on 
whether students were chronically absent in the fall is shown in Equation B.4: 


B.4. @71{ Pr(¥j; = 1)} = + ¢,Basic_ben; + ¢, Basic_cony + §7X{? + y; 


where Y;; is the attendance outcome for child i in block j such that Y,; equals 1 if the student was chronically 
absent and 0 if the student was not chronically absent. Basic_benj; is an indicator of whether child/family i in 


block j was assigned to the benefits-framed basic messaging and similarly Basic_conj;; is the indicator for whether 
(4) 
ij 
of student characteristics, and y; is a fixed effect for block j. &() is defined as the cumulative distribution 
function of the standard normal distribution. 


a child/family was assigned to the consequences-framed basic messaging during the fall semester, X;,’ is a vector 


To assess the impact of the basic messaging approaches relative to control, the study team tested whether ¢, or 
¢, is significantly different than zero using Wald tests. To test whether benefits- or consequences-framed basic 
messaging had different effects, the study team tested whether ¢, — ¢, was significantly different than zero. 


Analysis for Assessing the Added Effect of Intensified Messaging Over Basic Messaging 
Alone 


Using the SMART design, the study’s main impact analyses examined whether the basic messaging approaches 
had an impact on attendance outcomes in the fall, whether one intensified messaging approach had a larger 
impact than the other on attendance outcomes in the spring, and whether the combinations of basic and 
intensified messaging approaches had an impact on attendance across the year. The main analyses, however, did 
not address the question of whether adding intensified messaging for families of students with more absences in 
the fall made a difference, over and above continuing with basic messaging. This question is important because 
adding the intensified messaging also added to the cost of the messaging strategy. 
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The study was not designed to answer this question using the formal SMART analytic framework. That would 
have required assigning an additional “control” group during the second random assignment, in addition to the 
two intensified messaging groups. This control group would have been composed of families who were above 
the fall absence threshold, and they would have continued to receive only the basic messaging from the fall. 
Unfortunately, the study sample was not large enough to support this third group for the second random 
assignment. There were also ethical concerns about withholding intensified messaging from families who were 
“unresponsive” to basic messaging. 


To address this question, the study team instead used a regression discontinuity design (RDD) that took 
advantage of the fact that families were assigned to intensified messaging on the basis of a strict cutoff on fall 
absence rates (October to end of December 2017). Specifically, families that had at least one child who missed 8 
percent or more of school days in the fall were assigned to one of the two intensified approaches for the spring: 
School Staff Outreach or Goal Commitment messaging. Families that had all children below this threshold 
(missed less than 8 percent of school days in the fall) continued with basic messaging for the rest of the school 
year. 


A RDD provides valid causal inference when a continuous assignment variable and its cutoff score are used to 
assign units to intervention or comparison conditions (Shadish, Cook, & Campbell, 2002). The analysis yields an 
estimate of a local average treatment effect, or a measure of impact between those who fall immediately above 
and below the cut score. The analysis for RDD used a nonparametric model with two different bandwidths to 
estimate the effect of intensified messaging. The two algorithms that were used to select the bandwidths were IK 
(Imbens & Kalyanaraman, 2012) and CCT (Calonico, Cattaneo, & Titiunik, 2014). 


Regression discontinuity is a valid design to examine whether intensified messaging added to the effect of basic 
messaging, because families were assigned to receive an intensified messaging approach using a numerical 
forcing variable with a strict cutoff (8 percent), which was uniformly applied by the study team across all schools 
and families in the study, with no exceptions, making this a ‘sharp’ RDD. That is, all families in a basic messaging 
condition whose children were below the 8 percent cutoff continued to receive only basic messaging in the 
spring, while all families in a basic messaging condition who had at least one child above the 8 percent cutoff 
received both intensified messaging and basic messaging in the spring. 


The sections below present the steps taken to verify the validity of the design and analysis. These are aligned 
with the What Works Clearinghouse’s RDD standards (Version 4.0), which focus on four areas for sharp RDDs: (1) 
integrity of the forcing variable, (2) attrition, (3) continuity of the relationship between the forcing variable and 
the outcome, and (4) functional form and bandwidth. 


Integrity of the Forcing Variable. Because RDDs rely on a forcing variable to determine assignment to 
intervention and inferences about intervention effects, it is essential to evaluate whether the forcing variable 
could have been manipulated for certain families to affect their intervention assignment. Manipulation of the 
forcing variable was unlikely in this case because it would mean district or school staff changing raw, daily 
student attendance data in ways that would manipulate the aggregate-level forcing variable calculated by the 
study team—percentage of days absent between October and end of December 2017— to be above or below the 8 
percent cutoff, according to the intervention assignment preferred by the district or school staff for each 
student/family. The study team selected both the forcing variable and the 8 percent cutoff. And, based on the 
study team’s knowledge of participating districts’ attendance policies, the 8 percent cutoff did not correspond to 
any existing cutoff districts used for initiating attendance-related actions or interventions (for example, a phone 
call or letter home). Therefore, it is unlikely to be a cutoff on which district or school staff would already have 
been focused. Importantly, the study team did not communicate the 8 percent threshold to district and school 
staff who had access to the daily attendance data. Because these individuals had no knowledge of the threshold 
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and how it would be used to determine intervention conditions in the study, there was no plausible incentive or 
opportunity for manipulation around the 8 percent cutoff. 


Additionally, in establishing the integrity of the forcing variable, density plots did not reveal discontinuities at the 
8 percent threshold that are obviously larger than discontinuities observed at other points along the forcing 
variable. (The top panel of Exhibit B.26a shows the distribution for the overall sample used in the RDD analysis 
and the top panel of Exhibit B.26b shows the distribution for students with a history of high absences used in the 
RDD.) Because the forcing variable is based on discrete counts (percentage of days absent) over a relatively short 
2.5-month window, the forcing variable is unsurprisingly somewhat lumpy. This is not a concern for the validity 
of the RDD because as noted above, the lumpiness is not more pronounced at the 8 percent cutoff than at other 
parts of the distribution. This suggests that any discontinuities at the cutoff are naturally occurring and not due 
to manipulation. In fact, analogous density plots for the no-messaging control group (bottom panel of Exhibits 
B.26a and B.26b) show a very similar distribution. Since all families in the control group received no messaging 
for the entire school year, there was no incentive to manipulate their scores around the 8 percent cutoff. The 
similarity in these distributions therefore provides further evidence that there was unlikely to be manipulation. 


Finally, the study team examined the statistical integrity of the forcing variable by conducting a McCrary test to see 
if there are discontinuities in the forcing variable distribution at the RDD cutoff (McCrary, 2008). The McCrary test 
indicated a discontinuity in densities at the cutoff for the overall student sample (p = .01) but not for the subsample 
of students with a history of high absences (p = .41). A McCrary test on the no-messaging control group found a 
similar pattern of results, where there was a statistically significant discontinuity in the forcing variable distribution 
at the RDD cutoff for the overall sample (p = .04), but not for the subsample of students with a history of high 
absences (p = .75). Together, these results suggest that concerns about manipulation are likely to be unwarranted 
because of the clear institutional integrity of the forcing variable and the compelling evidence showing that the 
forcing variable is naturally lumpy. 
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Exhibit B.26a. Density plots for the forcing variables, intervention group (top) and no-messaging 
control group (bottom), overall student sample 
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Exhibit B.26b. Density plots for the forcing variables, intervention group (top) and no-messaging 
control group (bottom), subsample of students with history of high absences 
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Attrition. As in any experimental or quasi-experimental analysis, attrition in RDDs can bias inferences about 
intervention effects. For example, if the families who received intensified messaging had been likely to move out 
of a participating district, the effect of intensified messaging could be misestimated. To be included in the 
regression discontinuity analysis, as with the main analyses, a student must (1) not have been opted out of the 
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study by a parent, (2) have been enrolled for 100 percent of the school days, and (3) have a full set of 
demographic variables. Exhibit B.26c shows the number of students eligible for inclusion in the RDD analysis 
within the optimal bandwidth identified using the IK algorithm, and the percentage of these eligible students 
who were excluded from the final RDD analysis on chronic absences due to attrition. Students from families 
below the cutoff within the optimal bandwidth correspond to the “control” group, and students from families 
above the cutoff within the optimal bandwidth correspond to the “intervention” group. Attrition is reported for 
both the overall student sample and the subgroup of students with a history of high absences. The overall and 
differential attrition rates for both analytic samples correspond to a low level of attrition under the WWC’s 
“cautious” threshold. 


Exhibit B.26c. Attrition results, chronic absence 


Number of students 
eligible for inclusion in the | Number of students actually 


included in the RDD 
Below Above Overall | Differential 
cutoff cutoff attrition attrition 
Overall 5,575 2,948 5,140 2,621 8.9% 3.3% 
History of high absences 1,635 1,305 1,465 1,156 10.9% 1.0% 


SOURCE: Authors’ compilation. 


Continuity of the Relationship Between the Outcome and the Forcing Variable. The study team also examined 
the continuity of the relationship between the forcing variable and the outcome to examine the likelihood that 
something other than the addition of intensified messaging could explain impacts on chronic absence. The 
specified RDD functional form (i.e., the relationship between the forcing variable and the outcome) and optimal 
IK bandwidths were applied to examine baseline equivalence of covariates at the cutoff, including race, gender, 
disability status, English learner status, whether or not a family had a working cell phone at baseline, and the 
baseline indicator for history of high absences.*° That is, baseline equivalence tests were conducted on the same 
samples used in the RDD analysis of impacts on outcomes. 


The results of the baseline equivalence tests are shown in Exhibits B.26d (for the overall sample) and B.26e (for 
the subsample of students with a history of high absences). Since the outcome of interest in the RDD analysis is 
chronic absence, the primary baseline covariate of interest is a baseline measure of chronic absence or more 
specifically, the study’s indicator for whether the student has a “history of high absences” (chronically absent in 
the school year prior to the study or in the first month of the school year for the study, which was all prior to the 
start of the attendance messaging intervention). For the overall student sample, the magnitude of the difference 
at the cutoff in the likelihood of having a history of high absences was 0.12 standard deviation, which is within 
WWC?’s 0.25 upper limit. For the subsample of students with a history of high absences, this difference is by 
definition zero at the cutoff since all students in the analysis have a history of high absences. Tests of baseline 
equivalence on the other variables, such as race, disability status, English learner status, gender, and whether 
the student’s parent had a working cell phone at baseline, found most of these variables were within WWC’s 0.25 
limit for baseline differences expressed in standard deviation units. There were a few exceptions, most notably 


30 Following the WWC procedures for binary variables, results include Cox index values for each of the covariates. 

31 An alternative would be to use a baseline measure of student achievement. However, because the study sample is grades K 
through 5 elementary school students, only students in grades 4 and 5 would have baseline scores, because standardized 
testing usually does not begin until students are in grade 3. 
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that students below the cutoff were more likely to be Hispanic while students above the cutoff were more likely 
to be African American. All of these covariates were included as covariates in the RDD analyses. 


Analyses examining the continuity of the relationship between the outcome and the forcing variable also included 
scatterplots with regression estimates to describe the relationship between the binned forcing variable (i.e., the 
probability of chronic absence for each unique value of the forcing variable) and the outcome. These scatterplots are 
presented in the results section (see Exhibits C.3b and C.3d). These graphs do not obviously show any discontinuities 
at forcing variable values within the bandwidth that exceed two times the standard error of the impact estimated at 
the cutoff, which was 2.41 percent for the overall sample and 2.72 percent for the sample of students with a history of 
high absences. 


Exhibit B.26d. Baseline equivalence results, overall sample 


Estimated 
Predicted probability Predicted probability difference in std. 

Covariate basic messaging intensified messaging dev. units 
History of high absences 34.3% 29.8% -0.12 
African American 40.4% 58.1% 0.43 
Hispanic 38.5% 21.6% -0.50 
Student with disability 17.7% 15.8% -0.08 
English language learner 14.2% 13.7% -0.02 
White 15.5% 16.8% 0.06 
Female 49.4% 49.7% 0.01 
Working cell phone at baseline 83.5% 89.9% 0.34 


NOTE. Standardized differences for binary variables were estimated using the Cox index. Sample size = 5,140 students within the IK 
bandwidth, Basic messaging (under threshold); 2,621 students within the IK bandwidth, Intensified messaging (over threshold). 


Exhibit B.26e. Baseline equivalence results, sample of students with history of high absences 


Estimated 
Predicted probability Predicted probability | difference in std. 

Covariate basic messaging intensified messaging dev. units 
African American 39.6% 56.0% 0.40 
Hispanic 48.0% 27.9% -0.53 
Student with disability 22.5% 22.5% 0.00 
English language learner 20.4% 14.2% -0.26 
White 9.8% 11.4% 0.10 
Female 47.0% 46.0% -0.03 
Working cell phone at baseline 81.8% 86.6% 0.22 


NOTE: Standardized differences for binary variables were estimated using the Cox index. Sample size = 1,465 students within the IK 
bandwidth, Basic messaging (under threshold); 1,156 students within the IK bandwidth, Intensified messaging (over threshold). 


Finally, the study team conducted falsification tests to further examine the likelihood of observing 
discontinuities along different values of the forcing variable. Falsification tests are used to test for other 
discontinuities along the forcing variable that could partially explain the estimated intervention impact. The 
method involves identifying multiple cutoff values other than the enacted cutoff and testing for discontinuities 
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along the forcing variable. These analyses used the same RDD analysis model and IK bandwidth selection 
algorithm as the main RDD analysis, with four values below and five values above the enacted cutoff value of 8 
percent.*? The results of these analyses are presented in Exhibit B.26f. At one cutoff value below the enacted 
cutoff, 3.5 percent, and at one above, 14 percent, there was a statistically significant impact on the chronic 
absence outcome, for the overall student sample. Additionally, at one cutoff value just below the enacted cutoff, 
6.5 percent, there was a statistically significant impact on the chronic absence outcome for the students with a 
history of high absences. 


Exhibit B.26f. Falsification test results showing whether statistically significant effects on chronic 
absence in the spring exist at alternative cutoffs 


Students Students with a history 
overall of high absences (p- 
Cutoff value (EV) value) 
2.0% 0.64 0.77 
3.5% <0.001* 0.17 
5.0% 0.05 0.75 
6.5% 0.07 0.01* 
9.5% 0.91 0.25 
11.0% 0.76 0.69 
12.5% 0.06 0.48 
14.0% 0.01* 0.27 
15.5% 0.24 0.27 


NOTES: The cutoff value of 8 percent was used for the main RDD impact analysis, with results reported in Exhibits C.3a and C.3c. * p-value < 0.05. 


Functional form and bandwidth. Adequately specifying the relationship between the forcing variable and the 
outcome is essential in RDDs. Fortunately, several methods are available to help with identifying a functional 
range (i.e., bandwidth) around the cutoff where the forcing variable-outcome relationship is sufficiently linear. 
Moreover, these multiple approaches can be used to evaluate the robustness of the findings to how the 
bandwidth is selected. 


The confirmatory analysis used a nonparametric approach, a local linear regression model that accounts for 
differences in slopes above and below the cutoff (LLCHS). The analyses used two different bandwidth algorithms 
to test the robustness of the results. The local optimal bandwidth values for each contrast tested were 
established by using the Imbens and Kalyanaraman (2012) selection algorithm and the CCT bandwidth selection 
algorithm (Calonico, Cattaneo, & Titiunik, 2014). The local regression model was as follows: 


B.5. Yi = Bo + BiTi + BoaAVG + BsTYAVG + BaXy + BsSjt €4 


where Yj is the outcome for student i in school j; Ty is an indicator variable for whether the students’ family 
continued with basic messaging only or received additional intensified messaging, AV; is the assignment variable 
centered on 8 percent, Xjis a vector of centered baseline student characteristics, including indicators for race 
(African American, Native American, Asian, Hispanic, multiple races), special education status, English learner 
status, gender, an indicator for the student’s parent having a working mobile phone at baseline, and an indicator 


32 Minimum and maximum cutoff values were restricted so that at least 4 unique values of the forcing variable were on both 
sides of the enacted cutoff. 
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for having a history of high absences; and S; is a vector of school indicators. In addition to controlling for school 
fixed-effects, the analysis used cluster-robust standard errors at the assignment variable level, as implemented in 
the clubSandwich package in the R statistical software program. 


Analysis for Comparing the Intensified Messaging Approaches to Each Other 


The following model was used to estimate the relative effect of intensified messaging approaches (School Staff 
Outreach or Goal Commitment messaging) on chronic absence: 


B.6. Pr(¥jj=1) = (0) + C,INT, + ETXS? + yj) 


where Y;; is an indicator of whether student i in school j was chronically absent in the spring (Y,; equals 1 if the 
student was chronically absent and 0 if not), INT;; is an indicator for whether student/family i in school j was 
assigned to either School Staff Outreach or Goal Commitment (/NT;; equals 1 for School Staff Outreach and O for 
Goal Commitment), X - is a vector of student characteristics, and y; is a fixed effect for school j. ( ) is defined 
as the cumulative distribution function of the standard normal distribution. 


The analyses assessed whether one of the assigned intensified messaging approaches was more effective in the 
spring semester by testing whether ¢, is significantly different than zero using a Wald test. 


65 


APPENDIX C: DETAILED IMPACT AND IMPLEMENTATION FINDINGS 


Impact Analysis Results for Student Attendance 


This section provides more detail on the results presented in the main report about the impact of the text 
messaging approaches on chronic absence, as well as full results for analyses of the impact of the text messaging 
approaches on the number of days absent. Results for number of days absent are not presented in the main 
report because this was a secondary attendance outcome for the study. In this section we report results for both 
the chronic absence and the total number of days absent outcomes because readers may find it useful to view 
the results next to each other to facilitate interpretation. 


The study team estimated simple linear models for the number of days absent outcome, but also estimated more 
complex Poisson models as a sensitivity analysis. Poisson models explicitly treat the outcome as a non-negative 
count variable. The two modeling approaches yielded substantially similar results; that is, the models identified 
the same comparisons as statistically significant with similar p-values and effect sizes. The Poisson model results 
are included in Appendix E. 


The order of the presented results is as follows: 


1. Impact of the four adaptive interventions 
a. Impact of adaptive interventions on full-year chronic absence and number of days absent, for the overall 
sample and the subsample of students with a history of high absences 
b. Graphical presentation of the impact of adaptive interventions on full-year number of days absent 
c. Comparison of adaptive interventions to each other on full-year chronic absence and number of days 
absent, for the overall sample and the subsample of students with a history of high absences 
2. Impact of basic messaging approaches 
a. Impact of basic messaging approaches on chronic absence and number of days absent in fall 2017, for the 
overall sample and subsample of students with a history of high absences 
b. Graphical presentation of the impact of basic messaging approaches on number of days absent in fall 
2017, for the overall sample and subsample of students with a history of high absences 
c. Comparison of basic messaging approaches to each other on chronic absence and number of days absent 
in fall 2017, for the overall sample and subsample of students with a history of high absences 
3. Added benefit of intensified messaging to basic messaging 
a. Effects of any intensified messaging compared to any basic messaging on chronic absence in spring 2018 
for the overall sample and subsample of students with a history of high absences 
b. Graphical presentation of the results 
4. Comparison of intensified messaging approaches 
a. Impact of School Staff Outreach versus Goal Commitment Messaging on chronic absence and number of 
days absent in spring 2018 for the overall student sample, the subsample of students with a history of 
high absences, and the subsample of students whose home language was English 
b. Graphical presentation of the results 


Impact of the Adaptive Interventions 


Exhibits C.1a and C.1b show pairwise differences in full-year chronic absence and number of days absent, 
between each adaptive intervention and the no-messaging control condition, for the overall student sample and 
the subsample of students with a history of high absences. The results for full-year chronic absence support 
Exhibit 3 in the main report. Exhibit C.1c graphs pairwise differences between each adaptive intervention and 


66 


the no-messaging control condition on full-year number of days absent. Exhibits C.1d and C.le present the 
pairwise differences between each adaptive intervention on full-year chronic absence and number of days 
absent, for the overall student sample and the subsample of students with a history of high absences. Exhibits 
C.id and C.le present the evidence to support “The Different Approaches to Basic and Intensified Messaging 
Were Similarly Effective in Reducing Chronic Absence” section of the main report. 


Exhibit C.1a. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for all students (results displayed in Exhibit 2 in the main report) 


Messaging Estimated 
Outcome condition Control difference Effect size p-value 
Chronic absence (percentage) 
All vs. control 16.9 20.5 -3.6* -0.10 <0.001 
AI2 vs. control 18.1 20.5 -2.4* -0.07 0.001 
AI3 vs. control 17.6 20.5 -2.9* -0.08 <0.001 
AI4 vs. control 17.6 20.5 -2.9* -0.08 <0.001 
Number of days absent (mean) 
All vs. control 9.2 9.8 -0.6* -0.07 <0.001 
AI2 vs. control 9.2 9.8 -0.6* -0.07 <0.001 
AI3 vs. control 9.2 9.8 -0.6* -0.08 <0.001 
AI4 vs. control 9.2 9.8 -0.6* -0.08 <0.001 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Sample size = 108 schools; 6,631 students in AI1; 6,591 students in AI2; 6,583 students in AI3; 6,564 students in AI4; 7,862 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for history of 
high absences, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit C.1b. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for students with a history of high absences (results displayed in Exhibit 2 in the 
main report) 


Messaging Estimated 

COltiKweiits rxoyirentaceyny Control difference Effect size p-value 
Chronic absence (percentage) 

All vs. control 39.9 47.1 -7.2* -0.15 <0.001 
AI2 vs. control 42.2 47.1 -4.9* -0.10 0.008 
AI3 vs. control 39.8 47.1 -7.3* -0.15 <0.001 
AI4 vs. control 43.6 47.1 3.5 -0.07 0.061 
Number of days absent (mean) 

All vs. control 15.2 16.6 -1.4* -0.13 <0.001 
AI2 vs. control 15.4 16.6 -1.2* -0.11 0.001 
AI3 vs. control 15.3 16.6 -1.3* -0.12 <0.001 
AI4 vs. control 15.7 16.6 -0.9* -0.09 0.010 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Sample size = 108 schools; 1,509 students in All; 1,502 students in AI2; 1,497 students in AI3; 1,519 students in students in AI4; 2,088 students 
in control. 

Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

* Indicates that the average outcome for the adaptive intervention is statistically significantly different from control. The table displays 
original p-values. (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit C.1c. Pairwise differences between each adaptive intervention and the no-messaging control 
condition on full-year number of days absent 


Average Number of Days Absent 
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NOTES: Overall sample size = 108 schools; 6,631 students 
in All; 6,591 students in AI2; 6,583 students in AI3; 6,564 
students in AI4; 7,862 students in the no-messaging group. 
Sample size for students with a history of high absences = 
108 schools; 1,509 students in All; 1,502 students in AI2; 
1,497 students in AI3; 1,519 students in students in AI4; 
2,088 students in the no-messaging group. 

* Indicates that the average outcome for the adaptive 
intervention is statistically significantly different from the 
no-messaging group (p < .05) when the significance level is 
adjusted for four pairwise comparisons with a Bonferroni 
correction. 


Exhibit C.1d. Pairwise differences between each adaptive intervention, for all students 


Estimated 
Outcome All AI2 AI3 AI4 difference | Effect size | p-value 
Chronic absence (percentage) 
All vs. AI2 16.9 18.1 -1.2 -0.04 0.056 
All vs. AI3 16.9 17.6 -0.8 -0.02 0.290 
All vs. Al4 16.9 17.6 -0.7 -0.02 0.329 
AI2 vs. AI3 18.1 17.6 0.4 0.01 0.562 
AI2 vs. Al4 18.1 17.6 0.5 0.01 0.501 
AI3 vs. Al4 17.6 17.6 0.1 0.00 0.918 
Number of days absent (mean) 
All vs. AI2 9.2 9.2 0.0 0.00 0.886 
All vs. AI3 9.2 9.2 0.0 0.00 0.978 
All vs. Al4 9.2 9.2 0.0 0.01 0.739 
AI2 vs. AI3 9.2 9.2 0.0 0.00 0.878 
AI2 vs. Al4 9.2 9.2 0.1 0.01 0.640 
AI3 vs. AI4 9.2 9.2 0.0 0.01 0.722 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Sample size = 108 schools; 6,631 students in AI1; 6,591 students in AI2; 6,583 students in AI3; and 6,564 students in AI4. 

Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
Adaptive intervention group means were computed by adding the estimated differences between the adaptive interventions and the control 
condition to the control group mean (even though control group means not shown in this exhibit). 

The table displays original p-values. * Indicates that the average outcomes for any two adaptive interventions compared to each other are 
statistically significantly different from each other (p < .05), when the significance level is adjusted for six pairwise comparisons with a 
Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Exhibit C.1e. Pairwise differences between each adaptive intervention, for students with a history of 
high absences 


Estimated 
(Orin iis All AI2 AI3 AI4 difference | Effect size | p-value 
Chronic absence (percentage) 
All vs. AI2 39.9 42.2 -2.3 -0.05 0.181 
All vs. AI3 39.9 39.8 0.0 0.00 0.997 
All vs. Al4 39.9 43.6 -3.7 -0.08 0.068 
AI2 vs. AI3 42.2 39.8 2.3 0.05 0.249 
AI2 vs. Al4 42.2 43.6 1.4 -0.03 0.494 
AI3 vs. Al4 39.8 43.6 -3.7 -0.08 0.033 
Number of days absent (mean) 
All vs. AI2 15.2 15.4 -0.2 -0.02 0.664 
All vs. AI3 15.2 15.3 -0.1 -0.01 0.834 
All vs. Al4 15.2 15.7 -0.4 -0.04 0.279 
AI2 vs. AI3 15.4 15.3 0.1 0.01 0.852 
AI2 vs. Al4 15.4 15.7 -0.3 -0.03 0.492 
AI3 vs. Al4 15.3 15.7 -0.3 -0.04 0.333 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Sample size = 108 schools; 1,509 students in AI; 1,502 students in AI2; 1,497 students in AI3; and 1,519 students in AI4. 

Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Adaptive intervention group means were computed by adding the estimated differences between the adaptive interventions and the control 
condition to the control group mean (even though control group means not shown in this exhibit). 

The table displays original p-values. * Indicates that the average outcomes for any two adaptive interventions compared to each other are 
statistically significantly different from each other (p < .05), when the significance level is adjusted for six pairwise comparisons with a 
Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 


71 


Impact of Basic Messaging 


Exhibits C.2a and C.2b show impact estimates for the basic messaging approaches on chronic absence and 
number of days absent in the fall, for the overall student sample and subsample of students with a history of high 
absences. Exhibit C.2c graphs the results presented in Exhibits C.2a and C.2b for the number of days absent 
outcome. (Not displayed in the main report because number of days absent was a secondary attendance 
outcome measure for the study.) Finally, Exhibit C.2d shows the comparison of basic messaging approaches to 
each other for the overall student sample and subsample of students with a history of high absences. 


Exhibit C.2a. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for all students (results displayed in Exhibit 3 in the main report) 


Messaging Estimated 


Outcome condition (OXeyi laws) | difference Effect size | p-value 


Chronic absence (percentage) 


Any basic messaging vs. control 17.2 19.4 -2.2* -0.06 <0.001 
Benefits-framed basic vs. control 17.2 19.4 -2.2* -0.06 <0.001 
Consequences-framed basic vs. control 17.3 19.4 -2.1* -0.06 0.001 


Number of days absent (mean) 


Any basic messaging vs. control 2.7 2.9 -0.2* -0.08 <0.001 
Benefits-framed basic vs. control 2.7 2.9 -0.2* -0.08 <0.001 
Consequences-framed basic vs. control 2.7 2.9 -0.2* -0.07 <0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 7,665 students in benefits-framed basic messaging; 7,606 students in consequences-framed basic 
messaging; 7,862 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to end of December 2017. The 
analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at baseline 
indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction.** 


33 The significance levels were adjusted for four multiple comparisons (any basic messaging vs. control, benefits-framed basic 
vs. control, consequences-framed basic vs. control, and benefits- vs. consequences-framed basic messaging). The results 
comparing benefits-framed messaging to consequences-framed basic messaging is presented separately in Exhibit C.2d. 
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Exhibit C.2b. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for students with history of high absences (results displayed in Exhibit 3 in the 
main report) 


Messaging Estimated 
Outcome rxoyiveltaceyny (Keyra we) | difference Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 36.2 40.7 -4.5* -0.09 0.001 
Benefits-framed basic vs. control 36.3 40.7 -4.4* -0.09 0.005 
Consequences-framed basic vs. control 36.0 40.7 -4,.7* -0.10 0.003 
Number of days absent (mean) 
Any basic messaging vs. control 4.6 5.0 -0.4* -0.11 0.000 
Benefits-framed basic vs. control 4.6 5.0 -0.4* -0.12 0.000 
Consequences-framed basic vs. control 4.6 5.0 -0.4* -0.11 0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,064 students in benefits-framed basic messaging; 2,042 students in consequences-framed basic 
messaging; 2,088 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to end of December 2017. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit C.2c. Average number of days absent in fall 2017, by messaging group 


Average Number of Days Absent 


All students Students with a 
history of high absences 


Impact 
™ No messaging 
@ Benefits-framed basic messaging 


™@ Consequences-framed basic messaging 
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NOTES: Overall sample size = 108 schools; 7,665 students 
in benefits-framed basic messaging; 7,606 students in 
consequences-framed basic messaging; 7,862 students in 
the no-messaging group. 

Sample size for students with a history of high absences = 
108 schools; 2,064 students in benefits-framed basic 
messaging; 2,042 students in consequences-framed basic 
messaging; 2,088 students in the no-messaging group. 

* Indicates a statistically significantly difference between a 
messaging condition and the no-messaging group (p < .05) 
when the significance level is adjusted for four pairwise 
comparisons with a Bonferroni correction. 


Exhibit C.2d. Comparison of basic messaging approaches to each other for the overall student sample 
and subsample of students with a history of high absences 


Benefits- Consequences | Estimated 
Sample framed basic | -framed basic difference Effect size p-value 


Chronic absence (percentage) 


Overall student sample 17.2 17.3 -0.2 -0.01 0.748 
Students with a history of high 36.3 36.0 0.3 0.01 0.833 
absences 


Number of days absent (mean) 


Overall student sample 2.7 2.7 0.0 0.00 0.705 
Students with a history of high 4.6 4.6 -0.0 -0.01 0.772 
absences 


NOTES: Overall student sample = 108 schools; 7,665 students in benefits-framed basic messaging; 7,606 students in consequences-framed 
basic messaging. Subsample of students with a history of high absences = 108 schools; 2,064 students in benefits-framed basic messaging; 
2,042 students in consequences-framed basic messaging. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to end of December 2017. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Basic messaging group means were computed by adding the estimated differences between benefits-framed and consequences-framed basic 
messaging strategies and the control group to the unadjusted control group means. Control group means are shown in Exhibits C.2a and 
C.2b. 

The table displays original p-values. * Indicates that the average outcomes for the two basic messaging conditions (benefits- or consequences- 
framed) are statistically significantly different from each other (p < .05), when the significance level is adjusted for four pairwise comparisons 
with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Added Benefit of Intensified Messaging to Basic Messaging 


Exhibit C.3a shows results from the regression discontinuity analysis testing whether receiving any intensified 
messaging on top of basic messaging reduced spring chronic absence rates compared to only receiving any basic 
messaging, for the overall sample. Exhibit C.3b graphs the relationship between the forcing variable and 
probability of chronic absence for the overall sample within the IK bandwidth. Exhibits C.3c and C.3d show 
results for the same analysis for the subsample of students with a history of high absences. 


Exhibit C.3a. Percentage of students chronically absent in the spring who were from families above 
and below the 8 percent fall absence threshold, among students in the overall student sample (results 
displayed in Exhibit 5 in the main report) 


Chronic absence rate | Chronic absence rate Estimated 
for students with for students with basic difference 
intensified messaging messaging only (percentage 
Bandwidth (percentage) (percentage) points) Effect size 
IK 25.3 23.9 1.4 0.05 0.556 
CCT 20.6 20.7 -0.01 -0.01 0.085 


NOTES: Effect size is based on the Cox index for binary outcomes; IK = Imbens and Kalyanaraman; CCT = Calonico, Cattaneo, and Titiunik. 
The chronic absence rates reflect the predicted probabilities, at the cutoff, based on the confirmatory model using the IK algorithm. For 
example, predicted chronic absence rates were 23.9 percent for students from families just below the 8 percent cutoff whose parents were 
sent basic messaging only and 25.3 percent for students from families just above the 8 percent cutoff for students whose parents were sent 
basic messaging plus intensified messaging. Sample size = 5,140 students within the IK bandwidth, basic messaging; 2,621 students within the 
IK bandwidth, intensified messaging; 11,098 students within the CCT bandwidth, basic messaging; 3,357 within the CCT bandwidth, 
intensified messaging. 
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Exhibit C.3b. Graphical presentation of the continuity of the relationship between the forcing variable 
and the spring chronic absence outcome, for the overall student sample within the IK bandwidth 
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NOTES: Values reflect binned (i.e., simple average) values of the chronic absence outcome for each value of the forcing variable within the 
bandwidth. 
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Exhibit C.3c. Percentage of students chronically absent in the spring who were from families above 
and below the 8 percent fall absence threshold, among students in the subsample with a history of high 
absences (results displayed in Exhibit 4 in the main report) 


Chronic absence rate | Chronic absence rate Estimated 
for students with for students with basic Anivaraites 
intensified messaging messaging only (percentage 
Bandwidth (percentage) (percentage) points) Effect size 
Ik 34.5 40.1 -5.5 -0.14 0.042 
CCT 33.9 39.6 -5.7 -0.15 0.021 


NOTES: Effect size is based on the Cox index for binary outcomes; IK = Imbens and Kalyanaraman; CCT = Calonico, Cattaneo, and Titiunik. 
The chronic absence rates reflect the predicted probabilities, at the cutoff, based on the confirmatory model using the IK algorithm. For 
example, predicted chronic absence rates were 40.1 percent for students from families just below the 8 percent cutoff whose parents were 
sent basic messaging and 34.5 percent for students from families just above the 8 percent cutoff for students whose parents were sent basic 
messaging with intensified messaging. Sample size = 1,465 students within the IK bandwidth, basic messaging; 1,156 students within the IK 
bandwidth, intensified messaging; 1,654 students within the CCT bandwidth, basic messaging; 1,214 within the CCT bandwidth, intensified 
messaging. 
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Exhibit C.3d. Graphical presentation of the continuity of the relationship between the forcing variable 
and the spring chronic absence outcome, for the subsample of students with a history of high absences 
within the IK bandwidth 
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NOTES: Values reflect binned (i.e., simple average) values of the chronic absence outcome for each value of the forcing variable within the 
bandwidth. 
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Comparison of Intensified Messaging Approaches 


Exhibits C.4a, C.4b, and C.4c present results comparing the intensified messaging approaches to each other in 
spring 2018 for the overall student sample, the subsample of students with a history of high absences, and the 
subsample of students whose home language was English. Exhibits C.4a and C.4b support Exhibit 5 in the main 
report. Exhibit C.4d displays a graphical presentation of pairwise differences between School Staff Outreach and 
Goal Commitment messaging on number of days absent in spring 2018 (not shown in the main report), for the 
overall student sample, the subsample of students with a history of high absences, and the subsample of students 
whose home language was English. 


Exhibit C.4a. Impact estimates for the two intensified messaging approaches, compared to each other, 
on student attendance outcomes in the spring, for all students (results displayed in Exhibit 5 in the 
main report) 


School Staff Goal Estimated 
(Orin its Outreach Commitment difference | Effect size p-value 
Chronic absence (percentage) 43.7 46.8 3.1 -0.06 0.061 
Number of days absent (mean) 10.7 10.9 -0.2 -0.03 0.378 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,116 students in School Staff Outreach; 2,057 students in Goal Commitment messaging. 

Chronic absence and number of days absent in the spring are based on attendance records from January 2018 through end of the 2017-18 
school year. 

The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 


Exhibit C.4b. Impact estimates for the two intensified messaging approaches, compared to each other, 
on student attendance outcomes in the spring, for students with a history of high absences (results 
displayed in Exhibit 5 in the main report) 


School Staff Goal Estimated 
Outcome (O}iawervon) Commitment difference | Effect size p-value 
Chronic absence (percentage) 57.5 62.3 -4,8* -0.10 0.033 
Number of days absent (mean) 13.3 13.9 -0.6 -0.07 0.111 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 1,085 students in School Staff Outreach; 1,100 students in Goal Commitment messaging. 

Chronic absence and number of days absent in the spring are based on attendance records from January 2018 through end of the 2017-18 
school year. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 


Exhibit C.4c. Impact estimates for the two intensified messaging approaches, compared to each other, 
on student attendance outcomes in the spring (English home language subgroup analysis) 


School Staff (ever: | Estimated 
(OlTikeey rites Outreach Commitment difference | Effect size 
Chronic absence (percentage) 44.9 48.6 -3.7* -0.07 0.039 
Number of days absent (mean) 11.0 11.2 -0.2 -0.03 0.463 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 1,796 students in School Staff Outreach; 1,759 students in Goal Commitment messaging. 

Chronic absence and number of days absent in the fall are based on attendance records from January 2018 through end of the 2017-18 school 
year. 

The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 


Exhibit C.4d. Pairwise differences between School Staff Outreach and Goal Commitment messaging on 
number of days absent in spring, for the overall student sample, the subsample of students with a 
history of high absences, and the subsample of students whose home language was English 
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Sample of students in intensified messaging condition = 
108 schools; 2,116 students in School Staff Outreach; 2,057 
students in Goal Commitment messaging. 

Subsample of students with history of high absences = 108 
schools; 1,796 students in School Staff Outreach; 1,759 
students in Goal Commitment messaging. 

Subsample of students whose home language was English 
= 108 schools; 2,116 students in School Staff Outreach; 
2,057 students in Goal Commitment messaging. 
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Impact Analysis Results for Student Achievement 
This section presents the following results for the academic achievement outcome: 


Impact of adaptive interventions 


a. Impact of adaptive interventions on full-year reading and mathematics outcomes, for the overall sample and 
the subsample of students with a history of high absences (Exhibits C.5a and C.5b) 

b. Comparison of adaptive interventions to each other on full-year reading and mathematics outcomes, for the 
overall sample and the subsample of students with a history of high absences (Exhibits C.5c and C.5d) 


Exhibit C.5a. Pairwise differences between each adaptive intervention and control on achievement 
outcomes, for the overall student sample (results displayed in Exhibits 6 and 7 in the main report) 


Messaging Estimated 
Outcome condition Control ilitonontas Effect size 
Reading achievement 
All vs. control -0.067 -0.040 -0.027 -0.04 0.195 
AI2 vs. control -0.074 -0.040 -0.034 -0.05 0.102 
AI3 vs. control -0.054 -0.040 -0.014 -0.02 0.510 
AI4 vs. control -0.071 -0.040 -0.031 -0.04 0.132 


Mathematics achievement 


All vs. control -0.062 -0.055 -0.007 -0.01 0.734 
AI2 vs. control -0.067 -0.055 -0.012 -0.02 0.573 
AI3 vs. control -0.074 -0.055 -0.020 -0.03 0.348 
AI4 vs. control -0.079 -0.055 -0.025 -0.03 0.232 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Reading achievement sample size = 108 schools; 2,394 students in All; 2,365 students in AI2; 2,438 students in AI3; 2,439 students in AI4; 
2,787 students in control. Mathematics achievement sample size = 108 schools; 2,391 students in AI1; 2,360 students in AI2; 2,434 students in 
AI3; 2,435 students in AI4; 2,782 students in control. 

The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

Messaging group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit C.5b. Pairwise differences between each adaptive intervention and control on achievement 
outcomes, for students with a history of high absences (results displayed in Exhibits 6 and 7 in the 
main report) 


Messaging Estimated 
Outcome condition iitonones Effect size 
Reading achievement 
All vs. control -0.307 -0.321 0.014 0.02 0.742 
AI2 vs. control -0.301 -0.321 0.020 0.03 0.636 
AI3 vs. control -0.283 -0.321 0.038 0.05 0.369 
AI4 vs. control -0.323 -0.321 -0.002 -0.00 0.951 


Mathematics achievement 


All vs. control -0.349 -0.394 0.045 0.06 0.303 
AI2 vs. control -0.320 -0.394 0.074 0.10 0.094 
AI3 vs. control -0.369 -0.394 0.025 0.03 0.550 
AI4 vs. control -0.348 -0.394 0.046 0.06 0.264 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Reading achievement sample size = 108 schools; 501 students in AI1; 488 students in AI2; 569 students in AI3; 576 students in Al4; 706 
students in control. Mathematics achievement sample size = 108 schools; 500 students in AI; 486 students in AI2; 568 students in AI3; 574 
students in AI4; 702 students in control. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Messaging group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit C.5c. Pairwise differences between each adaptive intervention on academic outcomes, for the 
overall student sample 


Estimated 
Outcome All AI2 AI3 AI4 difference | Effect size | p-value 
Reading achievement 
All vs. AI2 -0.067 -0.074 0.007 0.02 0.600 
All vs. AI3 -0.067 -0.054 -0.013 -0.02 0.537 
All vs. Al4 -0.067 -0.071 0.004 0.01 0.855 
AI2 vs. AI3 -0.074 -0.054 -0.021 -0.03 0.345 
AI2 vs. AI4 -0.074 -0.071 -0.003 -0.00 0.881 
AI3 vs. Al4 -0.054 -0.071 0.017 0.04 0.197 
Mathematics achievement 
All vs. AI2 -0.062 -0.067 0.005 0.01 0.727 
All vs. AI3 -0.062 -0.074 0.012 0.02 0.574 
All vs. Al4 -0.062 -0.079 0.017 0.02 0.426 
AI2 vs. AI3 -0.067 -0.074 0.007 0.01 0.746 
AI2 vs. AI4 -0.067 -0.079 0.012 0.02 0.581 
AI3 vs. Al4 -0.074 -0.079 0.005 0.01 0.708 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Reading achievement sample size = 108 schools; 2,394 students in All; 2,365 students in AI2; 2,438 students in AI3; 2,439 students in AI4. 
Mathematics achievement sample size = 108 schools; 2,391 students in AI1; 2,360 students in AI2, 2,434 students in AI3; 2,435 students in AI4. 
The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

Adaptive intervention group means were computed by adding the estimated differences between the adaptive interventions and the control 
condition to the control group mean (even though control group means not shown in this exhibit). 

* Indicates that the average outcomes for any two adaptive interventions compared to each other are statistically significantly different from 
each other (p < .05), when the significance level is adjusted for six pairwise comparisons with a Bonferroni correction. 
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Exhibit C.5d. Pairwise differences between each adaptive intervention on academic outcomes, for 
students with a history of high absences 


Estimated 
Outcome All AI2 AI3 AI4 difference | Effect size | p-value 
Reading achievement 
All vs. AI2 -0.307 -0.301 -0.006 -0.01 0.872 
All vs. AI3 -0.307 -0.283 -0.024 -0.03 0.607 
All vs. Al4 -0.307 -0.323 0.017 0.02 0.710 
AI2 vs. AI3 -0.301 -0.283 -0.018 -0.02 0.705 
AI2 vs. Al4 -0.301 -0.323 0.023 0.03 0.610 
AI3 vs. Al4 -0.283 -0.323 0.040 0.07 0.262 
Mathematics achievement 
All vs. AI2 -0.349 -0.320 -0.029 -0.05 0.449 
All vs. AI3 -0.349 -0.369 0.020 0.03 0.660 
All vs. Al4 -0.349 -0.348 -0.000 -0.00 0.993 
AI2 vs. AI3 -0.320 -0.369 0.049 0.07 0.287 
AI2 vs. Al4 -0.320 -0.348 0.029 0.04 0.529 
AI3 vs. AI4 -0.369 -0.348 -0.021 -0.04 0.549 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Reading achievement sample size = 108 schools; 501 students in AI1, 488 students in AI2; 569 students in AI3; 576 students in AI4. 
Mathematics achievement sample size = 108 schools; 500 students in All; 486 students in AI2; 568 students in AI3; 574 students in AI4. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Adaptive intervention group means were computed by adding the estimated differences between the adaptive interventions and the control 
condition to the control group mean (even though control group means not shown in this exhibit). 

* Indicates that the average outcomes for any two adaptive interventions compared to each other are statistically significantly different from 
each other (p < .05), when the significance level is adjusted for six pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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IMPLEMENTATION OF THE TEXT MESSAGING STRATEGY 


This section includes descriptive statistics about the number of messages delivered and implementation fidelity 
and dosage. Exhibit C.6 provides information about the number of messages sent to parents in different 
messaging approaches. 


Exhibit C.6. Descriptive information about the number of messages sent, by message type 


Average 
number of 


Total number messages per | Standard 
messages sent # student 4 deviation | Minimum | Maximum 


Basic messaging only 


Weekly preventive message with tip 369,500 33.3 1.7 3 35 
Same-day absence notification 54,803 4.9 4.2 Oo 49 
Welcome back notification 43,036 3.9 3.3 O 26 
Total messages 467,339 42.1 7.7 5 100 
Intensified messaging: School 

Staff Outreach 

Weekly preventive message with tip 69,944 33.1 1.9 1 35 
Same-day absence notification 26,999 12.8 8.6 O 63 
Welcome back notification 20,220 9.6 6.0 O 34 
School Staff Outreach 7,055 3.3 4.2 O 63 
Total messages 124,218 58.7 15.2 1 130 
Intensified messaging: Goal 

Commitment messaging 

Weekly preventive message with tip 68,084 33.1 1.9 2 35 
Same-day absence notification 26,899 13.1 8.7 O 67 
Welcome back notification 20,360 9.9 6.2 0 49 
Goal Commitment Messages 42,155 20.5 1.2 1 22 
Goal Commitment feedback messages 41,197 20.0 1.9 1 22 
Total messages 198,695 96.6 15.7 7 193 


SOURCE: District administrative records and SchoolMessenger archived system data. 

NOTES: The analytic sample is students who were enrolled for 100 percent of the school days and had complete demographic data. Sample 
sizes are 11,097 students for the basic messaging only group; 2,115 students for the School Staff Outreach group; 2,057 students for the Goal 
Commitment messaging group. 

Although students were in the sample for the full school year, for some students a working SMS number was available only for a few days. 
This explains the low minimum number of weekly informational messages. 

4 The total and average number of messages per student were calculated to have at most one message per student counted each time a text 
was sent to parents of a student (i.e., multiple contact numbers were ignored). 


86 


Implementation fidelity was defined as the percentage of daily messages that were sent as intended (i.e., the 
correct message type was sent to the correct parents). For example, if a student was absent on Monday, parents 
in the benefits-framed basic messaging group should have received a same-day notification that day informing 
them that their child was absent, with information about the total number of absences to date during the school 
year, and a positively-framed benefit of regular attendance. 


Implementation dosage was defined as the percentage of correctly-sent messages that were recorded by the text 
messaging platform as successfully delivered to a local carrier. The fidelity measure does not capture whether 
the content of the message was correct (e.g., whether the number of absences to date was correct, as this 
information is based on each district’s daily student information system data). In addition, although the text 
messaging system allowed the study to see whether the text messages were delivered, the definition of dosage 
does not capture whether the text messages were actually received or read by parents. 


Rates of correct messaging were high across message types and intervention conditions. On average, parents in 
all messaging conditions were sent correct messages 94 percent of the days. Although text messages were sent 
with high levels of accuracy, rates of successful delivery were lower. Overall, about 70 to 75 percent of the total 
number of possible messages were recorded as delivered. Message delivery rates varied by message type. The 
dosage is lower than the accuracy rate because of non-functioning SMS numbers. (About 15 percent of families 
did not have functioning SMS numbers at baseline). 


Exhibit C.7 shows the average rates of correctly sent text messages and delivery rates for the pre-scheduled 
informational messaging and absence notifications sent to all families who were assigned to receive any text 
messages. 


Exhibit C.7. Messaging accuracy and delivery rate, by message type 


Average percent | Average percent 


of messages sent of messages 
correctly delivered 


Basic messaging only 


Weekly preventive message with tip 97.0 77.7 
Same-day absence notification 83.7 64.4 
Welcome back notification 86.9 66.9 
Total messages 94.9 75.6 


Intensified Messaging: School Staff Outreach 


Weekly preventive message with tip 97.5 74.9 
Same-day absence notification 86.7 63.4 
Welcome back notification 86.5 63.1 
School Staff Outreach 85.0 76.7 
Total messages 93.3 70.7 


Intensified messaging: Goal Commitment messaging 


Weekly preventive message with tip 96.3 72.5 
Same-day absence notification 88.4 62.5 
Welcome back notification 86.8 61.6 
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Average percent | Average percent 


of messages sent of messages 
correctly delivered 
Goal Commitment Messages 99.9 73.2 
Goal Commitment feedback messages 98.3 68.6 
Total messages 94.2 70.3 


SOURCE: District administrative records and SchoolMessenger archival data. 
NOTES: Sample size = 11,097 students for the basic messaging only group; 2,115 students for the School Staff Outreach group; 2,057 students 
for the Goal Commitment messaging group. 


Uptake of Intensified Messaging 


School Staff Outreach 


The purpose of the School Staff Outreach was to have a trained staff person, such as an attendance counselor, 
conduct personalized outreach to parents. The school staff assigned to conduct the outreach were asked to use 
the texting platform to send personalized messages to parents of students assigned to the School Staff Outreach 
messaging from the beginning of February through April, until parents responded by text or other modes. 


School staff conducted outreach as intended, based on two different sources of data. According to data from the 
text messaging system, school staff attempted to contact one or more parents of 85 percent of the students in the 
School Staff Outreach group. Similarly, attendance counselors reported contacting parents of 91 percent of the 
students who were included in their School Staff Outreach logs (Exhibit C.8). According to archived system data, 
30 percent of students had a parent who responded to at least one sent School Staff Outreach text message. Of 
the students whose parents received School Staff Outreach, 14.8 percent had a parent who responded to more 
than one sent text message (Exhibit C.9). However, these rates may underestimate the actual rate of response to 
School Staff Outreach because parents may have responded through other means than text (e.g. phone, in- 
person). 


Exhibit C.8. School Staff Outreach: percentage of students for whom contact was attempted, and time 
spent trying to contact or communicate with parents, by district and overall 


Percentage of students in Minutes spent texting or communicating with parents of students in 
Outreach group whose the School Staff Outreach group 


parents were sent at least | Average number Standard 
one school outreach text of minutes deviation Minimum Maximum 
1 92.3 14.1 19.6 0 104 
2 83.4 10.3 17.2 (0) 150 
3 94.3 15.5 21.2 0) 134 
4 95.4 11.8 19.3 (0) 147 
All districts 90.5 12.6 19.2 O 150 


SOURCE: Study attendance counselor logs. 
NOTES: Sample size = 938 students in the School Staff Outreach group, who were also included in the Attendance Counselor Logs and in the 
analytic sample. 
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Exhibit C.9. Percentage of students whose parents responded to School Staff Outreach through text 
messaging, by initial messaging approach, by district and overall 


Percent of students Percent of students | Percent of students 

whose parents were with a parent who with a parent who 

Basic messaging sent at least one responded to at least | responded to more 

District approach delivered text one sent text than one sent text 
1 Benefits 69.5 23.3 9.0 
Consequences 69.5 26.9 13.9 
2 Benefits 77.4 34.4 17.5 
Consequences 74.2 35.1 18.4 
3 Benefits 78.1 26.9 14.2 
Consequences 76.8 26.6 11.6 
4 Benefits 84.0 35.6 16.9 
Consequences 79.6 29.1 13.7 
All districts Benefits 78.1 31.1 15.0 
Consequences 75.3 29.8 14.7 


SOURCE: SchoolMessenger archived system data. 
NOTES: Sample size = 2,115 students in the analytic sample that were in the School Staff Outreach group. 


Goal Commitment Messaging 


Goal Commitment messaging included automated messages on Sundays asking parents to commit to a perfect 
attendance week and feedback on Fridays on student’s attendance that week. The Goal Commitment messaging 
also included cascading tips as part of the weekly Sunday informational messages. The tips addressed specific 
topics that changed by week, and parents could request more tips by texting a keyword. 


Parents of about 30 percent of students in the Goal Commitment messaging group committed to at least one 
perfect attendance week. Among those who committed at least once, the average number of weeks was about 
four (Exhibit C.10). Parents of about 10 percent of students in the Goal Commitment messaging group ever 
requested cascading tips. Parents who requested cascading tips requested about 2.5 tips, on average (Exhibit 
C.10). 
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Exhibit C.10. Percentage of students whose parents responded to Goal Commitment messaging, by 
basic messaging approach, overall and by district 


Average number of 
Percent of students | Average number of full Percent of cascading tips 
whose parent attendance weeks students requested, among 
Basic committed to at | committed, among parents | whose parents parents who 
messaging least one full who committed to at least requested requested at least one 
approach attendance week one week # cascading tips tip> 
1 Benefits 24.7 4.2 6.4 1.9 
Consequences 24.4 4.2 5.1 2.7 
2 Benefits 29.4 4.5 11.9 3.1 
Consequences 29.9 5.1 12.6 2.4 
3 Benefits 28.2 4.3 8.9 3.8 
Consequences 23.1 3.5 7.2 2.3 
4 Benefits 39.0 3.3 12.7 2.3 
Consequences 41.4 3.6 10.7 2.5 
All Benefits 31.0 4.0 10.3 2.8 
districts Consequences 30.9 4.1 9.3 2:5 


SOURCE: SchoolMessenger data archived system data. 
NOTES: Sample size = 2,057 families in the analytic sample and assigned to Goal Commitment messaging. * total of 642 parents committed to 
at least one week. » A total of 202 parents requested at least one tip. 


Description of Typical (Business-as Usual) Attendance-Related Outreach and Contrast 
in What Students in the Messaging vs. No-Messaging Groups Received 


Description of Business-as-Usual 


For districts and schools to be eligible to participate in the study, they could not already be systematically using 
text messaging to inform parents about attendance. However, it was known at the outset that multiple practices 
were already in place to improve attendance in study schools. The districts and schools were not expected to 
change their typical practices with the introduction of the text messaging strategy. Thus, the study’s text 
messaging approaches were layered over existing attendance policies, procedures, and practices - the “business- 
as-usual” services that the no-messaging group of families received. The business-as-usual condition provides 
context for understanding the impacts of the messaging strategy, since those effects are measured as the added 
benefit of the messaging on top of those existing, business-as-usual practices. This section summarizes 
information collected during the study about these existing practices in participating schools. 


According to the logs completed by school staff, most study schools (74 percent) had a formal attendance team. 
Most formal attendance teams had an attendance counselor/specialist (79 percent of the teams), the principal (61 
percent of the teams), and a school secretary (57 percent of the teams) as members (Exhibit C.11). Fewer teams 
(31 percent) had teachers as members of the formal attendance team. The most common attendance-related 
practices reported as in use by study schools were (Exhibit C.12): 

e Contacting parents by phone or mail when students were absent (85 percent) 

e Providing information to parents or guardians about the importance of attendance (85 percent) 


e Developing intervention plans with families and monitor for progress (73 percent) 
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Exhibit C.11. Business-as-usual attendance communication practices: Percentage of schools witha 
formal attendance team and composition of the team, overall and by district 


Percentage of Percentage of teams with... 


schools with Attendance 
formal Assistant specialist/ Yel iteye) | 
District attendance team | Principal principal counselor(s) | Teacher(s) | secretary 
1 79.2 73.7 47.4 89.5 47.4 73.7 42.1 
2 68.0 76.5 41.2 100.0 17.6 35.3 47.1 
3 87.0 35.0 20.0 65.0 40.0 45.0 70.0 
4 65.5 63.2 26.3 63.2 15.8 73.7 26.3 
All districts 74.3 61.3 33.3 78.7 30.7 57.3 46.7 


SOURCE: Study attendance counselor log. 
NOTES: Formal attendance team was defined as a team delegated by principals to lead the school’s efforts related to improving student 
attendance. Sample size = 100 schools completed the initial log including questions about business-as-usual practices. 


Exhibit C.12. Business-as-usual attendance communication practices: Percentage of schools indicating 
use of different attendance strategies 


Percentage of schools 


formally using 
Attendance strategy attendance strategies 


Tier I-type supports (for all students): 


Contact parents by phone or mail when students are absent 85.1 
Create a welcoming school climate 71.7 
Recognize good or improved attendance for classrooms, students, or parents 70.0 
Invite parents and community members to help address the barriers that keep children from 48.5 


attending school (e.g., health related issues, unsafe neighborhood) 


Tier II-type supports (for students in need of more support): 


Provide information to parents or guardians about the importance of attendance 85.1 
Assess individual student or family needs that affect attendance and contact parents or 84.2 
guardians as appropriate 

Establish specific attendance goals for students 67.3 
Involve public agencies and community partners and resources as needed to address barriers 66.0 
to attendance 

Have school staff identify barriers to attendance, such as health, transportation, or housing 65.7 
Involve the school nurse with follow-up on medical-related absences 62.6 


Tier III-type supports (for students in need of most intensive intervention) 


Develop intervention plans with families and monitor for progress 72.7 
Connect students with chronic physical and mental health issues to medical providers 66.7 
Make home visits and refer families to appropriate services (e.g., social services, human 66.0 
resources, housing, and health services) 

Use legal means (e.g., courts) to involve families and students in needed services when 63.9 
appropriate 
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Percentage of schools 


formally using 
Attendance strategy attendance strategies 


Arrange transportation to and from school with other families or staff 32.0 


SOURCE: Study attendance counselor logs. 
NOTES: Sample size = 100 schools that completed the initial log asking questions of business-as-usual practices. 


Service Contrast 


The attendance counselor logs asked the school staff to record whether they contacted parents, the amount of 
time spent contacting or trying to contact parents, the means of communication, and whether they referred 
parents to additional services or resources. The study collected this information for students whose parents 
received basic messaging only, basic messaging with School Staff Outreach or Goal Commitment, and for 
students whose parents did not receive any messaging (i.e., in the no-messaging control group).*4 


Data from the logs showed that parents of students with more absences received more frequent contact from 
school staff in all messaging groups. However, among parents of students with more absences, those in School 
Staff Outreach were contacted most often, as expected. The following patterns emerged, suggesting a link 
between frequency of contact and students’ absence level and the messaging approach (Exhibit C.13): 


e Parents of students with low absences (that is, missed less than 8 percent of school days in the fall) were 
contacted the least (52 to 53 percent of parents), regardless of basic messaging approach (benefits-framed, 
consequences-framed, or no-messaging). 


e Parents of students with higher absences (that is, missed more than 8 percent of school days in the fall) and 
who were in the Goal Commitment or no-messaging groups were contacted more often than parents of 
students with low absences (62 to 64 percent of parents). 


e Parents of students with higher absences and who were in the School Staff Outreach group were contacted 
most often (91 percent of parents). 


This pattern of school staff contacting parents of students with more absences most often when they were in the 
School Staff Outreach group is also reflected in the reported duration of time spent contacting parents and in the 
reported rates of parent referrals to services or other supports (a key purpose of School Staff Outreach): 


e On average, school staff reported spending about 13 minutes per student communicating with parents in 
School Staff Outreach. They reported spending about 8 minutes per student communicating with parents 
in the no-messaging group, who would have been eligible for an intensified intervention based on their 
child(ren)’s high fall absence rate (Exhibit C.13). 


e School staff were more than twice as likely to report using text messaging to contact parents of children in 
School Staff Outreach than parents of children in the no-messaging group, who would have been eligible 
for an intensified intervention based on their child(ren)’s high fall absence rate (about 66 percent versus 25 
percent) (Exhibit C.14). 


34 The main analyses of the amount of time spent contacting or trying to contact parents were conducted using linear 
regression models for simplicity. Poisson models were additionally used for sensitivity analyses because the number of 
minutes is a count variable. The results do not substantively differ (i.e., the models identify the same comparisons as 
statistically significant with similar p-values). 


92 


e School staff reported referring 41 percent of parents in School Staff Outreach to supports or services. They 
also reported referring 32 percent of parents in the no-messaging group, who would have been eligible for 
an intensified intervention based on their child(ren)’s high fall absence rate (Exhibit C.15). 


Together, these results suggest that School Staff Outreach was implemented as intended, and in particular, that 
there was a meaningful difference (as intended) in what those assigned to School Staff Outreach received and 
what those assigned to other groups received (including those who were in the no-messaging control group but 
were otherwise similar to those in School Staff Outreach in terms of fall absence rates). Thus, the impact findings 
presented earlier in Appendix C represent a meaningful test of the texting strategy. 


Exhibit C.13. Amount of time school staff reported spending trying to contact or communicate with 
parents, overall and by messaging condition 


Percentage of Minutes spent contacting or communicating 
students whose with parents of the sampled students 


Eligibility for intensified messaging and parents were | Average number | Standard 
messaging condition contacted of minutes deviation 


Below absence threshold (< 8 percent missed) 


Continuing basic messaging 51.7* 4.0* 8.8 0) 99 


No-messaging control group 52.7* 4.1* 8.0 0) 78 


Above absence threshold (2 8 percent missed) 


Basic messaging with School Staff Outreach 90.5 12.6 19.2 0) 150 
Basic messaging with Goal Commitment 62.4* 8.2* 16.9 0) 135 
No-messaging control group 64.0* 8.1* 16.9 0) 135 


SOURCE: Study attendance counselor logs. 

NOTES: Sample size = 106 schools participating in the study that completed at least one of the three follow up attendance counselor logs; 
3,947 students included in the attendance counselor logs and in the analytic sample. 

* Indicates that the school staff reported, on average, levels of contact with parents of students in each messaging or control condition that 
were statistically significantly different from the average levels of contact with parents of students in School Staff Outreach (p < .05). 
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Exhibit C.14. Percentage of attendance counselors communicating with students or their parents about 
attendance via different modes, overall and by messaging condition 


Percentage of attendance counselors 


In-person 
One-on-one or phone 


discussion | Emails or meeting 
Eligibility for intensified messaging and with writing Text with 
messaging condition student letters messaging parents 


Below absence threshold (< 8 percent missed) 


Continuing basic messaging 15.7* 21.3* 24.7* 30.3* 4.9* 


No-messaging control group 17.1* 22.7* 24.4* 33.6* 4.1* 


Above absence threshold (2 8 percent missed) 


Basic messaging with School Staff Outreach 29.5 32.6 65.6 51.3 7.4 
Basic messaging with Goal Commitment 22.2* 30.3* 28.3* 41.7* 8.5 
No-messaging control group 21.9* 28.8* 25.0* 42.5* 8.7 


SOURCE: Study attendance counselor logs. 

NOTES: 106 schools participating in the study that completed at least one of the three follow up attendance counselor logs; 3,947 students 
included in the attendance counselor logs and in the analytic sample. 

* Indicates that the likelihood of school staff reporting that they contacted parents of students in each messaging or control condition that 
were statistically significantly different from the likelihood of school staff reporting that they contacted parents of students in School Staff 
Outreach (p < .05), in different modes. 
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Exhibit C.15. Parent referrals to supports or services, according to school staff 


Percentage of Percentage of sample of oo whose pao were referred to 
students whose the following supports and services 


parents were Family 


Eligibility for intensified referred to District Breakfast/ | court/ 
messaging and messaging supports or student | Social lunch truancy 
condition services Transportation | services | services | programs | programs | Other 


Below absence threshold (< 8 percent missed) 


Continuing basic messaging 23.7* 5.4* 11.0 1.9* 6.7* 2.8* 6.0* 
No-messaging control group 23.8* 5.1* 10.7 2.3* 6.9% 2.8* 6.8* 
Above absence threshold (2 8 percent missed) 

Basic messaging with School 40.8 9.6 17.0 5.9 9.0 7.2 13.6 
Staff Outreach 

Basic messaging with Goal 33.7* 8.0* 12.6 4.3* 8.2 4.8* 9.6* 
Commitment 

No-messaging control group 32.2* 7.8* 12.5 5.2 8.2 5.2* 8.9* 


SOURCE: Study attendance counselor logs. 

NOTES: Sample size = 106 schools participating in the study that completed follow-up attendance counselor logs; 3,947 students included in 
the attendance counselor logs and in the analytic sample. 

* Indicates that the likelihood school staff reporting that they referred parents of students to supports or services in each messaging or 
control condition was statistically significantly different from the likelihood of reporting that they referred parents of students in School Staff 


Outreach (p < .05) to supports or services. 
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APPENDIX D: COST AND COST EFFECTIVENESS ANALYSES 


The study intended to test a relatively low-cost text messaging strategy that intensified for parents of students 
with more absences. A critical aspect of the study was to document the start-up and maintenance costs, to help 
prospective users consider what the costs might be in their own district or school. In addition, the study 
translated cost information for delivering the adaptive interventions into estimates of cost-effectiveness, taking 
into account the impact of the adaptive interventions on attendance outcomes. The following section describes 
the steps taken to assess the costs of the adaptive intervention. 


The study used the Resource Cost Model, an approach to cost analysis that involves organizing the data 
collection around the specific activities related to provision of a service, strategy, or an intervention. The method 
has its roots in the “ingredients” approach to cost analysis (Levin, 1983; Levin & McEwan, 2001); it models 
structure and “ingredients” of services as they are delivered. Spending variations can then be analyzed by the 
types, quantities, and prices of the different resources. This allows identification of the expected costs associated 
with replication and scaling up the intervention, as well as detail about the specific costs of different intervention 
components. 


The study constructed a Resource Cost Model for each participating district using the CostOut tool developed by 
the Center for Benefit-Cost Studies in Education at Teachers College, Columbia University 
(https://www.cbcse.org/costout). Data came from three sources: 


e Extant administrative data on expenditures for contracted services by SchoolMessenger and labor hours 
for the study team staff. 


e Interviews with the district IT and student information system departments about the time spent and any 
new software or hardware purchased for implementing the intervention. 


e Attendance counselor logs that asked school staff conducting the School Staff Outreach to document time 
spent contacting parents of students in School Staff Outreach condition, and the school or district resources 
to which they referred parents (results reported in Appendix C). 


In addition to these three sources, the study calculated the time school staff spent in trainings on the School Staff 
Outreach component to be included in the cost estimates. Exhibit D.1 lists the estimated costs for developing the 
text messaging content and platform. Exhibit D.2 shows the estimated implementation cost for the different text 
messaging components (i.e., basic messaging, School Staff Outreach, and Goal Commitment messaging). Exhibit 
D.3 shows the cost of delivering the text messaging approaches overall, as well as average cost per district, 
school, and student. 


After data were entered into the Resource Cost Model, the study team calculated the costs associated with the 
adaptive interventions. The study team then divided the total cost of each adaptive intervention by the total 
change in attendance that the adaptive intervention produced in order to provide the cost-effectiveness ratio for 
each adaptive intervention. For example, for each adaptive intervention, the “cost per additional student day of 
attendance” was calculated as (Total cost of the adaptive intervention)/(Number of students in the adaptive 
intervention*Days absent effect).*° Exhibit D.4 provides the estimated cost per unit of improvement. 


35 For All: Benefits-framed basic messaging with School Staff Outreach, the calculation would be $56,165/(6,631*0.60)= 
$56,165/3,972=$14.14 (see the first row of Exhibit D.4). 
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Exhibit D.1. Estimated text messaging development costs 


Cost ingredients Total cost 


Development 

Study team costs of message content development $92,037 
Texting provider costs for refining their main platform $58,540 
Texting provider costs for creating the outreach platform and additional tip functionality $44,200 
Total $194,777 


SOURCE: District administrative records and district interviews. 


Exhibit D.2. Estimated implementation costs per messaging component 


Components of the adaptive text messaging strategy Total cost 


Basic messaging $151,915 
Intensified messaging: School Staff Outreach $36,372 
Intensified messaging: Goal Commitment $15,045 
Total $202,612 


SOURCE: District administrative records and district interviews. 
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Exhibit D.3. Estimated text messaging implementation costs overall and average cost per district, 
school, and student 


Aspects of the adaptive text messaging strategy 
delivery All Per district Per school Per student 


Basic messaging (benefits- and consequences-framed) 


Study team costs 


Labor for programming messages $12,938 $12,938 $120 $0.13 

Labor for scheduling and monitoring $77,491 $19,373 $718 $3.00 
District costs 

Labor for setup and monitoring $22,338 $5,584 $207 $0.86 

Computers and software $7,448 $1,862 $69 $0.29 
Provider costs 

Labor for setup $23,900 $5,975 $221 $0.93 

Text message programming $7,800 $1,950 $72 $0.30 
Basic messaging total $151,915 $47,682 $1,407 $5.88 
School Staff Outreach 
Study team costs 

Labor for training $8,527 $2,132 $79 $3.51 
District Costs 

Labor for training $9,844 $2,461 $91 $4.05 

Labor for outreach $9,800 $2,450 $91 $4.03 
Provider Costs 

Labor for training $5,000 $1,250 $46 $2.06 

Text messages $3,200 $800 $30 $1.32 
School Staff Outreach total $36,372 $9,093 $337 $14.97 


Goal Commitment messaging 


Study team costs 


Labor for programming messages $3,234 $3,234 $30 $1.34 

Labor for scheduling and monitoring $8,610 $2,153 $80 $3.56 
Provider costs 

Text messages $3,200 $800 $30 $1.32 
Goal Commitment total $15,045 $6,187 $139 $6.27 


SOURCE: District administrative records and district interviews. 
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Exhibit D.4. Estimated adaptive intervention costs and cost per unit of improvement 


Cost of preventing a 


Cost per student from 

Cost per | additional student | becoming chronically 
Adaptive intervention Total cost | student | day of attendance absent 
All: Benefits-framed basic with School Staff $56,165 $8.47 $14.14 $235.28 
Outreach 
AI2: Benefits-framed basic with Goal $45,500 $6.90 $11.88 $287.65 
Commitment 
AI3: Consequences-framed basic with School $56,165 $8.53 $14.15 $294.20 
Staff Outreach 
AI4: Consequences-framed basic with Goal $45,500 $6.93 $10.70 $239.03 
Commitment 


SOURCE: District administrative records and district interviews. 
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APPENDIX E: SENSITIVITY, SUBGROUP, AND EXPLORATORY ANALYSES 


Sensitivity Analyses 


This section provides results of analyses conducted to check whether the main findings reported were sensitive 
to decisions about which students were included in analyses and different model specifications. 


As described in Appendix B, to be in the main analytic sample, students had to have: 


1. not been “opted out” of the study by their parents, meaning that when parents were informed about the 
study taking place in their school, they did not text or submit a paper form indicating that they did not want 
their child(ren) to participate*® 

2. been part of the original sample randomly assigned to basic messaging or the no-messaging control group in 
fall 2017 

3. been enrolled for 100 percent of the school days in the 2017-18 school year, which allowed the study team to 
create clear and consistent attendance outcome measures 

4. been included in district records with information available about their demographic characteristics 


This section reports results from analyses conducted on two alternative definitions of the analytic sample: 


1. students who were enrolled for at least 70 percent of school days 
2. all students in the basic messaging or no-messaging control groups at the beginning of the year regardless of 
missing data 


The model specifications for the sensitivity analyses testing different sample definitions are the same as for the 
main impact analyses described in Appendix C. For the purpose of brevity, this section presents results for 
analyses comparing basic messaging approaches and adaptive interventions to the no-messaging control 
condition, and the two intensified messaging approaches to each other. This section does not present results for 
analyses comparing adaptive interventions or basic messaging approaches to each other, but all the sensitivity 
analyses for these other SMART design comparisons were consistent with the main analyses. (That is, there were 
no statistically significant differences among the four adaptive interventions or between the two basic messaging 
approaches.) 


In addition to the sensitivity analyses with different sample definitions, this section includes results for the 
number of days absent outcome using Poisson and quasi-Poisson models. These analyses compared the four 
adaptive interventions to the control condition and each other, the basic messaging strategies to the control 
condition and each other, and the two intensified messaging approaches to each other. The results are 
substantially the same as the results from the linear models (i.e., the models identify the same comparisons as 
statistically significant, with similar p-values and effect sizes). 


Students Enrolled for at Least 70 Percent of School Days 


The reason the main analyses restricted the analytic sample to students who were enrolled for 100 percent of the 
school days was to avoid the problem of missing attendance data. Missing attendance data complicates the 


36 Most students who were opted out of the study by their parents were removed from the potential study sample prior to 
random assignment. Requests for opt out received after random assignment were still honored. Opt-out rates overall and by 
study condition are shown in Appendix B (Exhibits B.11 and B.12). Parents in the messaging groups were also able to 
“unsubscribe” so that a specific phone number would no longer receive the study’s text messages, but this did not remove a 
student from the study sample. 
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definition of the attendance outcome measures, including chronic absence (missing 10 percent or more of school 
days between October 1 and the end of the year) and number of days absent across that period. However, it is 
possible that this restriction of 100 percent enrollment was too stringent and excluded students who may have 
been more or less prone to respond to the messaging strategies than students enrolled for the full year. Including 
a wider range of students in the analysis might also make the estimated impacts more applicable to high-need 
districts, which typically have high rates of transfer out of the district during the year. 


To see whether the impact results were similar, one set of sensitivity analyses included a broader group of 
students - those who were enrolled in the study schools for at least 70 percent of school days. To calculate the 
attendance measures for this sample of students, the study team used the actual number of days enrolled. For 
example, if a student was enrolled for 120 days and was absent 12 days, the student would have missed 10 
percent of school days and considered to be chronically absent. This approach assumes that the rate of absences 
that students had during their enrollment would have remained constant if they had stayed enrolled for 100 
percent of school days. As a result, using the actual number of days enrolled as the denominator introduces 
some degree of error to the attendance measures. Nevertheless, the study analyses examined this alternative 
sample definition to check the sensitivity of the main results. 


Exhibits E.1a to E.1f present results for the overall sample and for the subsample of students with a history of 
high absences who were enrolled for at least 70 percent of the school days. These analyses compared (1) each 
adaptive intervention to no-messaging, (2) each basic messaging approach to no-messaging, and (3) the 
intensified messaging approaches to each other. 


The results from the analyses including students who were enrolled for at least 70 percent of the school days are 
consistent with the main results in all cases with one exception: 


e Analyses comparing the intensified messaging approaches to each other. The main impact analysis did 
not find a statistically significant difference between School Staff Outreach and Goal Commitment 
messaging for the overall student sample (though did for students with a history of high absences, in favor 
of School Staff Outreach). The sensitivity analyses with students who were enrolled for at least 70 percent 
of school days did reveal a statistically significant difference, in favor of School Staff Outreach, for both the 
overall sample and the subsample of students with a history of high absences. The larger difference 
between the two intensified approaches may have emerged because this sensitivity sample includes more 
mobile students, who tend to be more at-risk.*” Direct one-on-one outreach may therefore be a better 
strategy than Goal Commitment messaging for this broader, more vulnerable group. 


37 Green, DeFossett, & Kuo, 2019. 
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Exhibit E.1a. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for students overall 


Messaging Estimated 


(OlTineey site reroyaebtateynl (Oeyrtawey| difference Effect size 


Chronic absence (percentage) 


All vs. control 17.4 21.2 -3.8* -0.10 <0.001 
AI2 vs. control 18.8 21.2 -2.4* -0.06 0.001 
AI3 vs. control 18.3 21.2 -2.9* -0.08 <0.001 
AI4 vs. control 18.2 21.2 -3.0* -0.08 <0.001 


Number of days absent (mean) 


All vs. control 9.4 10.0 -0.6* -0.07 <0.001 
AI2 vs. control 9.4 10.0 -0.6* -0.07 <0.001 
AI3 vs. control 9.4 10.0 -0.5* -0.07 <0.001 
AI4 vs. control 9.3 10.0 -0.6* -0.08 <0.001 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed basic messaging; where “if need” is defined as missing 8 percent of days or more between the 
start of October and the end of December 2017. 

Sample size = 108 schools; 6,788 students in AI1; 6,759 students in AI2; 6,763 students in AI3; 6,740 students in AI4; 8,088 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for history of 
high absences, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Exhibit E.1b. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for students with a history of high absences 


Messaging Estimated 


(OlTineey site reroyacebtaceynl (Oeyrtawey| difference Effect size 


Chronic absence (percentage) 


All vs. control 40.8 47.9 -7.2* -0.14 <0.001 
AI2 vs. control 43.2 47.9 -4.8* -0.10 0.009 
AI3 vs. control 41.1 47.9 -6.9* -0.14 <0.001 
AI4 vs. control 44.6 47.9 -3.3 -0.07 0.067 


Number of days absent (mean) 


All vs. control 15.4 16.7 -1.3* -0.12 <0.001 
AI2 vs. control 15.5 16.7 -1.2* -0.11 0.001 
AI3 vs. control 15.5 16.7 -1.2* -0.11 0.001 
AI4 vs. control 15.9 16.7 -0.8 -0.08 0.023 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed basic messaging; where “if need” is defined as missing 8 percent of days or more between the 
start of October and the end of December 2017. 

Sample size = 108 schools; 1,575 students in All; 1,573 students in AI2; 1,576 students in AI3; 1,596 students in AI4; 2,188 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Exhibit E.1c. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for students overall 


Messaging Estimated 

Outcome condition Control difference | Effect size p-value 
Chronic absence (percentage) 

Any basic messaging vs. control 17.7 19.9 2:2* 0.06 <0.001 
Benefits-framed basic vs. control 17.6 19.9 2.3* 0.06 <0.001 
Consequences-framed basic vs. control 17.8 19.9 2.1* 0.06 <0.001 
Number of days absent (mean) 

Any basic messaging vs. control 2.8 3.0 0.2* 0.07 <0.001 
Benefits-framed basic vs. control 2.8 3.0 0.2* 0.08 <0.001 
Consequences-framed basic vs. control 2.8 3.0 0.2* 0.07 <0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 7,876 students in benefits-framed basic messaging 7,839 students in consequences-framed basic 
messaging; 8,088 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to end of December 2017. 

The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction.*® 


38 The statistical significance levels were adjusted for four multiple comparisons (any basic messaging vs. control, benefits- 
framed basic vs. control, consequences-framed basic vs. control, and benefits- vs. consequences-framed basic messaging). 
The results comparing benefits-framed basic messaging to consequences-framed basic messaging for the impact analyses 
were presented in Appendix C. The differences between benefits- and consequences-framed basic messaging were not 
statistically significant for any of the main impact analysis, nor for any of the sensitivity analyses conducted. Thus, these 
results are not presented here in Appendix E. 
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Exhibit E.1d. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for students with a history of high absences 


Messaging Estimated 
Outcome condition Control difference | Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 36.8 41.3 -4,5* -0.09 0.001 
Benefits-framed basic vs. control 36.9 41.3 -4.4* -0.09 0.005 
Consequences-framed basic vs. control 36.6 41.3 -4.7* -0.10 0.003 
Number of days absent (mean) 
Any basic messaging vs. control 4.7 5.1 -0.4* -O.11 <0.001 
Benefits-framed basic vs. control 4.7 5.1 -0.4* -0.12 <0.001 
Consequences-framed basic vs. control 4.7 5.1 -0.4* -0.11 0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,166 students in benefits-framed basic messaging; 2,161 students in consequences-framed basic 
messaging; 2,194 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to end of December 2017. 

The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 


Exhibit E.1e. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for students overall 


School Staff Goal Estimated 
(Orin its Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 44.7 48.0 -3.3* -0.07 0.037 
Number of days absent (mean) 10.9 11.1 -0.2 -0.03 0.327 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,216 students in School Staff Outreach; 2,164 students in Goal Commitment messaging. 

Chronic absence and number of days absent are based on attendance records from January 2018 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for history of 
high absences, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 
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Exhibit E.1f. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for students with a history of high absences 


School Staff Goal Estimated 
(Orin its Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 58.6 63.4 -4.8* -0.10 0.033 
Number of days absent (mean) 13.5 14.1 -0.6 -0.07 0.111 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 1,149 students in School Staff Outreach; 1,1168 students in Goal Commitment messaging. 

Chronic absence and number of days absent are based on attendance records from January 2018 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 


All Students at the Start of the Study in a Basic Messaging or No-Messaging Control 
Condition, with Multiple Imputation to Account for Missing Data 


The main analyses included students who were enrolled for 100 percent of the days and not missing baseline 
demographic data. Although the level of missing data in the study was low (see Exhibit E.2), it is possible that the 
results could be different if students who did not meet these criteria were included in the analytic sample. For 
example, students who transferred out of the district may have been more at-risk than students who remained 
for the full year, and more at-risk students may have been affected by the texting strategy differently than less at- 
risk students. To examine this possibility, the study conducted sensitivity analyses using the entire sample of 
students initially in the basic text messaging or no-messaging conditions who were not opted out by parents. 
Because this sample included students who were missing data for one or more variables, multiple imputation 
was used to fill in the missing information. 


Exhibit E.2. Rates of missing data 


Variables Percent of total sample 


Missing at least one demographic baseline variable 1.33 
Missing attendance data for at least one day in first quarter 2.15 
Missing attendance data for at least one day in second quarter 3.53 
Missing attendance data for at least one day in third quarter 4.76 
Missing attendance data for at least one day in fourth quarter 6.97 


SOURCE. Authors calculations. 
NOTES. Sample size = 26,843 originally in the initial messaging conditions or no-messaging control group. 


Multiple imputation is a Monte Carlo technique in which missing values are replaced by m>1 simulated versions. 
In this study, m was set to 10 (Little & Rubin, 1987; Rubin 1987; Schafer 1997; Schafer & Graham, 2002). Multiple 
imputation is a valid method for handling missing data when the missing data mechanism is ‘missing at random,’ 
or MAR. In the case of MAR, missing patterns can be predicted by observed covariates (Rubin, 1987). This 
assumption implies that missing data does not depend on unobserved covariates, after controlling for observed 
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ones.*° Missing data analyses conducted with the study data showed that multiple baseline characteristics 
predicted whether students in the study had missing data, including indicators for being White, Hispanic, having 
a history of high absences, being an English language learner, and having a parent with a working cell phone 
number at baseline. These results suggest that the use of multiple imputation is a reasonable approach for 
handling missing data in this study. 


The study team used the R-mice package to conduct the multiple imputation. This study applied a time-ordered 
nested conditional imputation model to impute missing data, as it is the only approach used for SMART designs 
thus far (Shortreed et al., 2014). This approach is cascading in the sense that the imputed and observed values 
from the baseline are used as predictors in an imputation model for missing information in the fall, and the 
observed and imputed values from the baseline and the fall are then used as predictors in the imputation model 
for missing information in the spring. Missing baseline data were imputed first using only baseline data. This was 
done using a fully conditional specification of each of the baseline variables. First, the missing baseline data were 
imputed by filling in the missing values for a variable by sampling at random from the data that were observed 
for that variable. The first variable to be imputed was denoted as X,, while X;,...,X, denoted the additional 
baseline variables. For the baseline variables that were binary (such as gender), a logistic regression model of the 
form: 


E. 1. logit{Pr(X, = 1| Xo, ...,X_~ )} = Gy + AgXg t+ aX, 


was fit to the data for which X, was observed. The missing values of X, were then imputed by sampling from the 
posterior predictive distribution of X,. Then the missing values for variable X, were imputed using the same 
model conditioned on X,, X3, ...,X, such that the imputed values from the previous step were used for the missing 
values in X,. This process was repeated for all of the variables to complete one cycle. The same process was used 
to cycle through the rest of the variables repeatedly until the results stabilized and one imputed data set of the 
baseline variables was constructed. 


Next, the missing data for the percentage of days absent in the fall were imputed. The fall semester was split in 
half so that the first half of the semester ran from October 1 to November 12. Those who were not enrolled for the 
entire period were considered to be missing. Let X = (X,,...,X,) bea vector of baseline covariates and Y, be the 
percentage of days absent during the first half of the fall. The imputation model for the percentage of days absent 
was a regression model that included (1) the imputed baseline variables, (2) the intervention indicators, and (3) 
interactions between the intervention indicators and baseline covariates: 


E. 2. Y; = ay + 2X + 3A, + a4A,X 


As before, the model was fit to the data for which Y, is observed, and the missing values were imputed by 
sampling from the posterior predictive distribution. The model was fitted repeatedly with the sample drawn 
from the posterior distribution until the results stabilized. The percentage of days absent during the second half 
of the fall was imputed next, where those students who were not enrolled for the entire second half were 
considered to have missing data. Let Y, denote the percentage of days absent during the second half of the fall. 
The same procedure was used to impute the missing data using a model that included both the baseline imputed 
data and the imputed data for the first half of the fall semester, including the (1) imputed baseline variables, (2) 
intervention indicators, (3) the interactions between intervention indicators and baseline variables, (4) the 
imputed percentage of days absent for the first part of the fall semester, and (5) the interactions between the 
intervention indicators and the percentage of days absent in the first part of the fall semester: 


39 The MAR assumption, however, cannot be formally tested, nor is it possible to test whether missingness depends on the 
values that are missing (the ‘not missing at random’ or NMAR assumption). 
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E.3. Y= ay +a@2X + a3A,+ ayA,X + a5Y, + AY. 


After the imputation was completed for the fall semester, the missing data for the spring semester that also 
included the additional intensified messaging strategies were imputed. The first step was to impute the 
intervention status for students who were missing an intervention assignment for the spring based on their 
imputed absences in the fall. Those students from families who were below the 8 percent threshold continued to 
have the same intervention status as during the fall and those students from families who were above the 8 
percent threshold were randomized to School Staff Outreach or Goal Commitment messaging with equal 
probability. Otherwise the percentage of days missed was imputed similarly to the fall semester. The spring 
semester was split into two, from January 2 to April 1 and from April 2 to the end of the Spring semester. Let Y, 
and Y, denote the percentage of days absent during the first and second half of the spring, respectively. A 
regression approach similar to the one used for imputing missing fall data was used to impute the missing spring 
data, with a model that included (1) the baseline observed and imputed data, (2) fall intervention indicators, (3) 
the interactions between fall intervention indicators and baseline variables, (4) the interactions between spring 
intervention indicators and baseline variables, (5) the interaction between fall and spring intervention indicators, 
(6) the imputed percentage of days absent for the first and second parts of the fall semester, and (7) the 
interactions between fall and spring intervention indicators and the percentage of days absent in the first and 
second parts of the fall semester: 


E.4. Y3 = a@ + @2X +a3A, + @4A,X + AsA2X + aAgAzA, + a7 VY + agAy¥ + agA2Y 
where Y = (Y,,Y>) 


The same approach was used for the second half of the spring. The imputation model is the same, other than the 
addition of the percentage of days absent in the first part of the spring semester, and the interactions between 
fall and spring intervention indicators and the percentage of days absent in the first part of the spring semester: 


E.5.  Y,= a, +a,X +a@3A,+ a@4A,X + asA,X + a¢A,A, +a, Y + agA¥Y + aoAzY 
where Y = (Yj, Y, Y3) 


All of these procedures were repeated 10 times to create 10 multiple imputed data sets. The percentage of days 
absent in the fall and the indicator for whether a student was chronically absent in the fall was then calculated 
from each of the resulting data sets. The overall estimate is the average of individual estimates from each 
imputed data set. The standard errors were adjusted for within- and between-imputation variance. 


The results of these analyses conducted on the entire student sample that were in families randomly assigned at 
the start of the study (and that did not subsequently opt out) are shown in the exhibits that follow. Exhibits E.3a 
and E.3b present attendance outcomes for the overall student sample and the subsample of students with a 
history of high absences, comparing each adaptive intervention to the no-messaging control condition. Exhibits 
E.3c and E.3d present attendance outcomes for the overall sample and students with a history of high absences, 
comparing each basic messaging condition to the no-messaging control condition. Exhibit E.3e and E.3f present 
the attendance outcomes for the overall sample and students with a history of high absences, comparing the two 
intensified messaging approaches to each other. 
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The results from the analysis using the full sample with multiple imputation to account for missing data are 
consistent with the main results in almost all cases. The few exceptions noted below largely come from analyses 
examining the study’s secondary measure of attendance, which was number of days absent.*° 


Analyses comparing adaptive interventions to the control condition. The main impact analysis found 
significant differences between each adaptive intervention and the no-messaging control for both chronic 
absence and number of days absent outcomes for the overall sample and for students with a history of high 
absences. The analysis using multiple imputation indicated similar significant results for the chronic 
absence outcome for the overall sample and the subsample of students with a history of high absences. 
However, unlike the main impact analysis, for the number of days absent outcome only one of the adaptive 
interventions had a statistically significant impact (the one that assigned consequences-framed basic 
messaging at baseline and intensified with Goal Commitment messaging) for the overall sample, and none 
of the adaptive interventions had significant effects on the number of days absent outcome for students 
with a history of high absences. 


Analysis comparing basic messaging approaches to the control condition. The main impact analyses 
found both benefits- and consequences-framed basic messaging reduced both chronic absence and the 
number of days absent, compared to the no-messaging control for the overall sample and students with a 
history of high absences. The analysis using multiple imputation did not find statistically significant 
differences between the individual comparisons of benefits- or consequences-framed basic messaging to 
the control group for the chronic absence outcome for the overall sample. (However, like the main impact 
analysis, the analysis using multiple imputation found a significant impact for any basic messaging on the 
chronic absence outcome when compared to the no-messaging control condition.) For students with a 
history of high absences, the analysis using multiple imputation was consistent with the main analyses - 
both benefits- and consequences-framed messages reduced chronic absence and the number of days 
absent, compared to the no-messaging control condition. 


Analysis comparing intensified messaging approaches. In the main impact analyses, chronic absence 
rates and number of days absent for the overall sample appeared lower for students in School Staff 
Outreach than Goal Commitment messaging, but the difference was not statistically significant. This 
difference was statistically significant for students with a history of high absences. With the entire sample 
with multiple imputation, number of days absent (but not chronic absence rates) were statistically 
significantly lower for students overall in School Staff Outreach than those in Goal Commitment messaging; 
and among students with a history of high absences, School Staff Outreach was a statistically significantly 
better intensified strategy than Goal Commitment messaging, in terms of reducing both chronic absence 
and the number of days absent. 


40 Multiple imputation is the only recommended missing data approach for SMART designs but has been only recently 
applied in SMART analyses. The study team used the one approach that has been best-documented in the research literature: 
a time-ordered nested conditional imputation model. As the application of multiple imputation to SMART designs is relatively 
new and research comparing different missing data approaches for SMART designs is lacking, it is unclear why the results for 
the number of days absent outcome differ for the multiple-imputed sample and the main analysis sample. 
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Exhibit E.3a. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, full student sample with multiple imputation to address missing data 


Messaging Estimated 


(Oltinweyiites recoyaentaceynl (Oeyrtawe)| difference Effect size 


Chronic absence (percentage) 


All vs. control 14.9 18.0 -3.1* -0.08 <0.001 
AI2 vs. control 15.7 18.0 -2.3* -0.06 0.001 
AI3 vs. control 16.0 18.0 -2.0* -0.05 0.005 
AI4 vs. control 15.6 18.0 -2.4* -0.07 <0.001 


Number of days absent (mean) 


All vs. control 9.7 10.1 -0.4 -0.02 0.145 
AI2 vs. control 9.6 10.1 -0.5 -0.03 0.050 
AI3 vs. control 9.8 10.1 -0.3 -0.02 0.281 
AI4 vs. control 9.5 10.1 -0.6* -0.05 0.002 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging; where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December 2017. 

Sample size = 108 schools; 7,346 students in AI1; 7,338 students in AI2; 7,306 students in AI3; 7,284 students in AI4; 8,782 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for history of 
high absences, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive interventions is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit E.3b. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for students with history of high absences with multiple imputation to address 
missing data 


Messaging Estimated 


COlTineey rites reroyaentaceynl difference Effect size 


Chronic absence (percentage) 


All vs. control 41.7 47.8 -6.1* -0.12 <0.001 
AI2 vs. control 42.8 47.8 -5.0* -0.10 0.008 
AI3 vs. control 42.0 47.8 -5.8* -0.12 0.001 
AI4 vs. control 44.2 47.8 -3.6 -0.07 0.040 


Number of days absent (mean) 


All vs. control 16.0 16.8 -0.8 -0.04 0.244 
AI2 vs. control 15.7 16.8 -1.1 -0.05 0.136 
AI3 vs. control 16.1 16.8 -0.7 -0.03 0.304 
AI4 vs. control 16.0 16.8 -0.8 -0.05 0.133 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging; where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December 2017. 

Sample size = 108 schools; 1,764 students in All; 1,766 students in AI2; 1,758 students in AI3; 1,772 students in AI4; 2,442 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit E.3c. Impact estimates for the comparison of basic messaging approaches on chronic absence 
and number of days absent in the fall, for the full student sample with multiple imputation to address 
missing data 


Messaging Estimated 
COltiKweiits key eleitaceyyy Control difference | Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 15.9 18.2 -2.3* -0.06 <0.001 
Benefits-framed basic vs. control 15.9 18.2 -2.3 -0.06 0.055 
Consequences-framed basic vs. control 15.8 18.2 -2.4 -0.06 0.050 
Number of days absent (mean) 
Any basic messaging vs. control 2.9 3.1 -0.2* -0.11 <0.001 
Benefits-framed basic vs. control 2.9 3.1 -0.2* -0.07 <0.001 
Consequences-framed basic vs. control 2.9 3.1 -0.2* -0.06 <0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 8,566 students in benefits-framed basic messaging; 8,486 students in consequences-framed basic 
messaging; 8,782 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to the end of December 2017. The 
analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at baseline 
indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit E.3d. Impact estimates for the comparison of basic initial messaging approaches on chronic 
absence and number of days absent in the fall, for students with history of high absences with multiple 
imputation to address missing data 


Messaging Estimated 
COltiKweyiits key eteitaceyyy (Xo rtawe) | difference | Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 36.5 41.3 -4.8* -0.10 <0.001 
Benefits-framed basic vs. control 36.4 41.3 -4,9* -0.10 0.001 
Consequences-framed basic vs. control 36.5 41.3 -4,.8* -0.10 0.001 
Number of days absent (mean) 
Any basic messaging vs. control 4.8 5.2 -0.4* -0.12 <0.001 
Benefits-framed basic vs. control 4.8 5.2 -0.5* -0.10 <0.001 
Consequences-framed basic vs. control 4.9 5.2 -0.4* -0.09 0.002 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,408 students in benefits-framed basic messaging; 2,376 students in consequences-framed basic 
messaging; 2,436 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to the end of December 2017. The 
analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 


Exhibit E.3e. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for the full student sample with multiple imputation to 
address missing data 


School Staff Goal Estimated 
Outcome Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 48.9 51.8 -3.0 0.06 0.069 
Number of days absent (mean) 16.9 17.1 -0.3* 0.06 0.036 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,594 students in School Staff Outreach; 2,580 students in Goal Commitment messaging. 

Chronic absence and number of days absent are based on attendance records from January 2018 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Exhibit E.3f. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for students with a history of high absences with multiple 
imputation to address missing data 


School Staff Goal Estimated 
Outcome Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 64.5 69.0 -4,5* 0.10 0.030 
Number of days absent (mean) 20.2 20.7 -0.5* 0.09 0.015 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 1,139 students in School Staff Outreach; 1,348 students in Goal Commitment messaging. 

Chronic absence and number of days absent are based on attendance records from January 2018 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, working SMS at baseline indicator, and a set of student 
covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 


114 


Alternative Model Specification for the Number of Days Absent Outcome 


For simplicity, the main impact analyses for the number of days absent outcome were conducted using linear 
models. However, because the number of days absent is a count variable, we also conducted sensitivity analyses 
using a Poisson model for comparing the adaptive interventions to the no-messaging control and to each other, a 
quasi-Poisson model for comparing the basic messaging approaches to the no-messaging control and to each 
other, and a quasi-Poisson model comparing the two intensified messaging strategies to each other.*! These 
alternative models explicitly account for the fact that number of days absent is a non-negative count outcome. 
Exhibits E.4a, E.4b, E.4c, and E.4d present the results from these analyses for students overall. The results are 
similar to those for the main analyses (that is, the same comparisons are statistically significant, with similar p- 
values and effect sizes.) 


Exhibit E.4a. Pairwise differences between each adaptive intervention and control on number of days 
absent over the full year, for students overall 


Messaging Estimated 


Outcome reroyaKertacoyel (Oeyrtaweyt difference Effect size 


Number of days absent (mean) 


All vs. control 8.4 8.9 -0.6* -0.08 <0.001 
AI2 vs. control 8.4 8.9 -0.5* -0.08 <0.001 
AI3 vs. control 8.4 8.9 -0.5* -0.07 <0.001 
AI4 vs. control 8.4 8.9 -0.6* -0.08 <0.001 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed basic messaging; where “if need” is defined as missing 8 percent of days or more between the 
start of October and the end of December 2017. 

Sample size = 108 schools; 6,788 students in AI1; 6,759 students in AI2; 6,763 students in AI3; 6,740 students in AI4; 8,088 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a Poisson regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for 
history of high absences, and a set of student covariates. 

Control group means are unadjusted means; messaging group means were computed by adding the estimated differences to the control 
group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 


“| In general, the quasi-Poisson model was preferred to the Poisson model because the absence data are over-dispersed due to 
a large number of Os. However, the more complex analyses involving the full-year absence data and various adaptive 
interventions (Exhibits E.4a and E.4b) were run using a Poisson model because of convergence problems with the quasi- 
Poisson model. Since those analyses were estimated in the Generalized Estimating Equation framework, standard errors were 
corrected for distributional violations. 
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Exhibit E.4b. Pairwise differences between each adaptive intervention on number of days absent over 
the full year, for students overall 


Estimated 

Outcome All AI2 AI3 AI4 difference | Effect size p-value 
Number of days absent (mean) 

All vs. AI2 8.4 8.4 -0.0 -0.01 0.760 
All vs. AI3 8.4 8.4 -0.0 -0.01 0.771 
All vs. Al4 8.4 8.4 -0.0 -0.00 0.938 
AI2 vs. AI3 8.4 8.4 0.0 0.00 0.975 
AI2 vs. Al4 8.4 8.4 0.0 0.00 0.851 
AI3 vs. Al4 8.4 8.4 0.0 0.00 0.807 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging. Where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December 2017. 

Sample size = 108 schools; 6,631 students in AI1; 6,591 students in AI2; 6,583 students in AI3; 6,564 students in AI4. 

Number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 

The analyses are based on a Poisson regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for 
history of high absences, and a set of student covariates. 

The table displays original p-values. * Indicates that the average outcomes for any two adaptive interventions compared to each other are 
statistically significantly different from each other (p < .05), when the significance level is adjusted for six pairwise comparisons with a 
Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 


Exhibit E.4c. Impact estimates for the basic messaging approaches on number of days absent in the 
fall, for students overall 


Messaging Estimated 


COlTikeeyiites rxoyerelin(oyel difference Effect size 


Number of days absent (mean) 


Any basic messaging vs. control 2.1 2.9 -0.2* -0.08 <0.001 
Benefits-framed basic vs. control 2.7 2.9 -0.2* -0.08 <0.001 
Consequences-framed basic vs. control 2.7 2.9 -0.2* -0.07 <0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 7,876 students in benefits-framed basic messaging; 7,839 students in consequences-framed basic 
messaging; 8,088 students in control. 

Number of days absent in the fall are based on attendance records from October 1 to the end of December 2017. The analyses are based on a 
quasi-Poisson regression model controlling for school fixed effects, working SMS at baseline indicator, indicator for history of high absences, 
and a set of student covariates. 

Control group means are unadjusted means; messaging group means were computed by adding the estimated differences to the control 
group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit E.4d. Impact estimates for the two intensified messaging approaches, compared to each other, 
on number of days absent outcome in the spring, for students overall 


School Staff Goal Estimated 
Outcome Outreach Commitment difference | Effect size p-value 
Number of days absent (mean) 10.8 11.0 -0.2 -0.03 0.378 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 2,116 students in School Staff Outreach; 2,057 students in Goal Commitment messaging. 

Number of days absent is based on attendance records from January 2018 through the end of the 2017-18 school year. 

The analyses are based on a quasi-Poisson regression model controlling for school fixed effects, indicator for history of high absences, 
working SMS at baseline indicator, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means.* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal 
Commitment messaging) are statistically significantly different from each other (p <.05). 


Subgroup Analyses 


The study hypothesized that the text messaging might be more effective for students with a prior history of high 
absences and prioritized examining findings for that subsample of students alongside the overall sample. The 
study also conducted exploratory analyses for three other subgroups of students. (The student samples used in 
these analyses were subsets of the overall student sample used for the main impact analyses - that is, students 
who were not opted out of the study and were enrolled for 100 percent of school days, with complete 
demographic data.) 


1. District subgroup: these analyses excluded one district (out of four participating districts total) that 
experienced the greatest degree of messaging delivery challenges. These analyses included a total of 79 
schools and 16,569 students in the three other districts. 

— The study team hypothesized that the attendance messaging may have been more effective when 
excluding the one district where messages were more often not sent as planned. 


2. Home language subgroup: these analyses included only students for whom the home language indicated in 
the district student information system was English, a total of 18,691 students. 

— The study team hypothesized that the attendance messaging may have been more effective when 
students whose parents received messages in languages other than English are excluded. The reason 
for this hypothesis was that although the messages were carefully and professionally translated into 
many languages, they were developed in English. 


3. Subgroup of students with both a history of high absences and a working mobile phone at baseline: These 
analyses included only students who had both a history of high absences (missed 10 percent or more of 
school days in the prior year or in the first month of the 2017-18 school year) and had at least one parent with 
a working cell phone number in the first month of the 2017-18 school year. As reported in Appendix B, 27 
percent of students in the study sample had a history of high absences, and 85 percent had a parent witha 
working phone at baseline; 22 percent had both, or a total of 5,193 students in this subgroup. 

— This subgroup of students was arguably the group that might be expected to experience the greatest 
impact of the adaptive text messaging strategy; that is, students who were at risk for chronic absence 
due to previous attendance patterns and whose parents were more likely to be reachable via text 
messaging. 


For brevity, this section does not present subgroup results for analyses comparing adaptive interventions or 
basic messaging approaches to each other, but all the subgroup analyses for these comparisons were consistent 
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with the main analyses. (That is, there were no statistically significant differences among the adaptive 
interventions or between the two basic messaging approaches.) 


The subgroup analyses did not reveal any new patterns of results. The effects of the adaptive interventions and 
basic messaging approaches, and the comparison of intensified messaging approaches were similar to those 
found in the main impact analyses. 


District Subgroup Analysis 


Exhibit E.5a shows results for analyses comparing each adaptive intervention to the no-messaging control condition 
from the three districts without message delivery problems. Exhibit 5.b shows results for analyses comparing each 
basic text messaging approach to no-messaging, and Exhibit 5.c shows results for analyses comparing the two 
intensified text messaging approaches to each other, for the overall sample in the district subgroup. 


These results are similar to those from the main impact analyses reported in Appendix C with the following 
exception, which appears to possibly be due to reduced statistical power with the smaller sample size, because 
the point estimate is about the same or larger in magnitude, but the difference is not statistically significant: 


e Analysis comparing basic messaging approaches to the no-messaging control condition. The 
comparison between consequences-framed basic messaging and the control condition is not statistically 
significant for the chronic absence outcome. 
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Exhibit E.5a. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for the overall student sample in the three districts included in the district 
subgroup analysis 


Messaging Estimated 


(OlTixeey rite reroynentaceyel difference Effect size 


Chronic absence (percentage) 


All vs. control 17.6 20.8 -3.2* -0.09 0.000 
AI2 vs. control 18.8 20.8 -2.2* -0.06 0.009 
AI3 vs. control 17.4 20.8 -3.4* -0.09 0.000 
AI4 vs. control 17.4 20.8 -3.4* -0.10 0.000 


Number of days absent (mean) 


All vs. control 9.7 10.3 -0.6* -0.07 0.000 
AI2 vs. control 9.8 10.3 -0.6* -0.07 0.000 
AI3 vs. control 9.6 10.3 -0.7* -0.09 0.000 
AI4 vs. control 9.7 10.3 -0.7* -0.08 0.000 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging; where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December 2017. 

Sample size = 79 schools; 4,748 students in All; 4,743 students in AI2; 4,735 students in AI3; 4,710 students in AI4; 5,656 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcomes for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit E.5b. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for the overall student sample in the three districts included in the district 
subgroup analysis 


Messaging Estimated 
COltixweyiits rxoyelertaceyyy Control difference | Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 16.7 18.5 -1.8* -0.05 0.003 
Benefits-framed basic vs. control 16.6 18.5 -1.9* -0.06 0.005 
Consequences-framed basic vs. control 16.9 18.5 -1.6 -0.05 0.021 
Number of days absent (mean) 
Any basic messaging vs. control 2.8 3.0 -0.2* -0.07 0.000 
Benefits-framed basic vs. control 2.8 3.0 -0.2* -0.08 0.000 
Consequences-framed basic vs. control 2.8 3.0 -0.2* -0.07 0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 79 schools; 5,474 students in benefits-framed basic messaging; 5,439 students in consequences-framed basic 
messaging; 5,656 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to the end of December 2017. The 
analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at baseline 
indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 
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Exhibit E.5c. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for the overall student sample in the three districts 
included in the district subgroup analysis 


School Staff Goal Estimated 
Outcome Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 45.1 48.5 -3.4 -0.07 0.090 
Number of days absent (mean) 11.5 11.9 -0.4 -0.05 0.213 


SOURCE: District administrative records. 

NOTES: Sample size = 79 schools; 1,460 students in School Staff Outreach; 1,430 students in Goal Commitment messaging. 

Chronic absence and number of days absent in the spring are based on attendance records from January 2018 through end of the 2017-18 
school year. The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working 
SMS at baseline indicator, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 
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Home Language Subgroup Analysis 


Exhibit E.6a shows results for analyses comparing each adaptive intervention to the no-messaging control 
condition among students whose home language was English. Exhibit E.6b shows results for analyses comparing 
each basic text messaging approach to no messaging, and Exhibit E.6c shows results for analyses comparing the 
two intensified text messaging approaches to each other, for the overall sample in the English home language 
subgroup. 


These results are similar to those of the main impact analyses reported in Appendix C with the following 
exception, which again, seems possibly due to reduced statistical power with the smaller sample size: 


e Analysis comparing basic messaging approaches to the control condition. The comparison between 
consequences-framed basic messaging and the control condition is not statistically significant for the 
chronic absence outcome. 


Exhibit E.6a. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for students in the overall sample whose home language was English 


Messaging Estimated 


(Oltixeoyiites condition difference Effect size 


Chronic absence (percentage) 


All vs. control 17.8 21.9 -4.1* 0.11 0.000 
AI2 vs. control 19.4 21.9 -2.5* 0.07 0.002 
AI3 vs. control 19.2 21.9 -2.7* 0.07 0.001 
AI4 vs. control 19.0 21.9 -2.9* 0.08 0.000 


Number of days absent (mean) 


All vs. control 9.7 10.2 -0.6* 0.07 0.000 
AI2 vs. control 9.7 10.2 -0.5* 0.07 0.000 
AI3 vs. control 9.7 10.2 -0.5* 0.06 0.001 
AI4 vs. control 9.7 10.2 -0.6* 0.07 0.000 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging; where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December 2017. 

Sample size = 108 schools; 5,267 students in All; 5,235 students in AI2; 5,267 students in AI3; 5,262 students in AI4; 6,398 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at 
baseline indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Exhibit E.6b. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for students in the overall sample whose home language was English 


Messaging Estimated 
Outcome condition Control difference | Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 18.4 20.3 -1.9* -0.05 0.001 
Benefits-framed basic vs. control 18.1 20.3 -2.2* -0.06 0.002 
Consequences-framed basic vs. control 18.6 20.3 -1.7 -0.05 0.015 
Number of days absent (mean) 
Any basic messaging vs. control 2.8 3.0 0.2* 0.06 0.000 
Benefits-framed basic vs. control 2.8 3.0 0.2* 0.07 0.000 
Consequences-framed basic vs. control 2.9 3.0 0.2* 0.06 0.001 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 6,143 students in benefits-framed basic messaging; 6,150 students in consequences-framed basic 
messaging; 6,398 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to the end of December 2017. The 
analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working SMS at baseline 
indicator, and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 


Exhibit E.6c. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for students in the overall sample whose home language 
was English 


School Staff Goal Estimated 
CO)rixeny iis Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 44.9 48.6 -3.7* -0.07 0.039 
Number of days absent (mean) 11.0 11.2 -0.2 -0.03 0.463 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 1,796 students in School Staff Outreach; 1,759 students in Goal Commitment messaging. 

Chronic absence and number of days absent in the spring are based on attendance records from January 2018 through end of the 2017-18 
school year. The analyses are based on a regression model controlling for school fixed effects, indicator for history of high absences, working 
SMS at baseline indicator, and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means.* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal 
Commitment messaging) are statistically significantly different from each other (p < .05). 


Subgroup of Students with A History of High Absences and Working Mobile Phone at Baseline 


These analyses by definition focused on the students with a history of high absences, further subset by those 
who had at least one parent with a working mobile phone number prior to the start of the study. Exhibit E.7a 
presents results from analyses comparing each adaptive intervention to no messaging; Exhibit E.7b presents 
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results from analyses comparing the basic text messaging approaches to no messaging; and Exhibit E.7c presents 
results from analyses comparing the intensified text messaging approaches to each other. 


These results are similar to the main impact analysis results with the sample of students with a history of high absences, 
except for the following which again, seem possibly due to reduced statistical power with a smaller sample size: 


e Analysis comparing intensified messaging strategies. In the main impact analyses, chronic absence rates 
and number of days absent for the subsample of students with a history of high absences were statistically 
significantly lower for students in School Staff Outreach than Goal Commitment messaging. For the 
subgroup of students with a history of high absences and working mobile phone at baseline, the results 
show a similar pattern but are not statistically significant. 


Exhibit E.7a. Pairwise differences between each adaptive intervention and control on full-year 
attendance outcomes, for students with a history of high absences and a working phone at baseline 


Messaging Estimated 


(OlTixeey rites condition (Oeyvtaweyt difference Effect size 


Chronic absence (percentage) 


All vs. control 37.8 46.6 -8.8* -0.18 0.000 
AI2 vs. control 39.7 46.6 -6.9* -0.14 0.001 
AI3 vs. control 37.8 46.6 -8.8* -0.18 0.000 
AI4 vs. control 41.9 46.6 -4.7 -0.10 0.020 


Number of days absent (mean) 


All vs. control 14.7 16.5 -1.9* -0.17 0.000 
AI2 vs. control 15.0 16.5 -1.5* -0.14 0.000 
AI3 vs. control 14.9 16.5 -1.7* -0.16 0.000 
AI4 vs. control 15.2 16.5 -1.4* -0.13 0.000 


SOURCE: District administrative records. 

NOTES: All assigns benefits-framed basic messaging at baseline, then School Staff Outreach if need, otherwise continue with benefits-framed 
messaging; AI2 assigns benefits-framed basic messaging at baseline, then Goal Commitment messaging if need, otherwise continue with 
benefits-framed messaging; AI3 assigns consequences-framed basic messaging at baseline, School Staff Outreach if need, otherwise continue 
with consequences-framed messaging; AI4 assigns consequences-framed basic messaging at baseline, Goal Commitment messaging if need, 
otherwise continue with consequences-framed messaging; where “if need” is defined as missing 8 percent of days or more between the start 
of October and the end of December. 

Sample size = 108 schools; 1,274 students in AI1; 1,270 students in AI2; 1,251 students in AI3; 1,289 students in AI4; 1,756 students in control. 
Chronic absence and number of days absent are based on attendance records from October 1 through the end of the 2017-18 school year. 
The analyses are based on a regression model controlling for school fixed effects and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for days absent); messaging group means 
were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average outcome for the adaptive intervention is statistically significantly different 
from control (p < .05) when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 
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Exhibit E.7b. Impact estimates for the basic messaging approaches on chronic absence and number of 
days absent in the fall, for students with a history of high absences and a working phone at baseline 


Messaging Estimated 
(Orin its condition (Xenia we) | difference | Effect size p-value 
Chronic absence (percentage) 
Any basic messaging vs. control 34.4 40.7 -6.3* -0.13 0.000 
Benefits-framed basic vs. control 34.9 40.7 -5.8* -0.12 0.001 
Consequences-framed basic vs. Control 34.0 40.7 -6.7* -0.14 0.000 
Number of days absent (mean) 
Any basic messaging vs. control 4.5 5.0 -0.6* -0.15 0.000 
Benefits-framed basic vs. control 4.5 5.0 -0.5* -0.14 0.000 
Consequences-framed basic vs. control 4.4 5.0 -0.6* -0.15 0.000 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 1,732 students in in benefits-framed basic messaging; 1,705 students in students in consequences-framed 
basic messaging; 1,756 students in control. 

Chronic absence and number of days absent in the fall are based on attendance records from October 1 to the end of December 2017. The 
analyses are based on a regression model controlling for school fixed effects and a set of student covariates. 

Control group means are unadjusted mean percentages (for chronic absence) or unadjusted means (for the days absent outcome); messaging 
group means were computed by adding the estimated differences to the control group means. 

The table displays original p-values. * Indicates that the average messaging (any basic messaging, benefits-framed basic messaging, 
consequences-framed basic messaging) student attendance outcome is statistically significantly different from the average control student 
attendance outcome (p < .05), when the significance level is adjusted for four pairwise comparisons with a Bonferroni correction. 

The estimated difference does not always equal the difference between the means of compared conditions due to rounding. 


Exhibit E.7c. Impact estimates for the intensified messaging approaches, compared to each other, on 
student attendance outcomes in the spring, for students with a history of high absences and a working 
phone at baseline 


School Staff Goal Estimated 
Outcome Outreach Commitment | difference | Effect size | p-value 
Chronic absence (percentage) 55.8 60.2 -4.4 -0.09 0.079 
Number of days absent (mean) 12.9 13.5 -0.6 -0.07 0.116 


SOURCE: District administrative records. 

NOTES: Sample size = 108 schools; 878 students in School Staff Outreach; 912 students in Goal Commitment messaging. 

Chronic absence and number of days absent in the spring are based on attendance records from January 2018 through end of the 2017-18 
school year. The analyses are based on a regression model controlling for school fixed effects and a set of student covariates. 

School Staff Outreach means are unadjusted means; Goal Commitment means were computed by adding the estimated differences to the 
School Staff Outreach means. 

* Indicates that the average outcomes for the two intensified messaging conditions (School Staff Outreach, Goal Commitment messaging) are 
statistically significantly different from each other (p < .05). 
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Quantile Regression Analyses 


Exploratory analyses examined whether the basic messaging approaches - which on average reduced chronic 
absence both overall and for students with a history of high absences - had different effects for students with 
relatively low or relatively high absences in the fall. This information sheds light on whether the basic messaging 
approaches worked because they affected attendance only for students with a large number of absences, only for 
students with relatively few absences, or because they affected attendance for students more universally. These 
results are only suggestive, as they only focus on the effect of the basic messaging in the fall, the time between 
the first and second round of random assignment. The basic messaging effect in the fall is the focus because a 
quantile regression approach has not yet been developed for SMART designs that covers the intervention and 
outcomes over the entire school year. The quantile regression approach also focuses on the number of days 
absent outcome rather than the chronic absence outcome because quantile regression methods are substantially 
more developed for continuous outcomes than for binary outcomes. Exhibit E.8a shows the number of students, 
mean, standard deviation, minimum, and maximum for the days absent outcome for each quantile (defined as 
deciles for this analysis), for the overall student sample. 


Exhibit E.8a. Descriptive statistics for deciles 


Mean number of Standard 


days absent deviation Minimum Maximum 
0-10.0% 2,313 0.0 0.0 O 0 
10.1-20.0% 2,313 0.0 0.0 O 0 
20.1-30.0% 2,313 0.7 0.4 O 1 
30.1-40.0% 2,314 1.0 0.0 1 1 
40.1-50.0% 2,313 1.7 0.4 1 2 
50.1-60.0% 2,313 2.1 0.3 2 3 
60.1-70.0% 2,314 3.0 0.1 3 3.5 
70.1-80.0% 2,313 41 0.3 3.5 5 
80.1-90.0% 2,313 5.6 0.6 5 7 
90.1-100.0% 2,314 9.8 3.3 7 36 
All Students 23,133 2.8 3.1 0 36 


SOURCE: District administrative records. 


The quantile regression analyses estimated whether the effect of basic messaging varied for students located in 
different parts of the distribution of number of days absent in the fall. A quantile regression models the 
relationship between a set of predictor variables and specific quantiles (or deciles in this case) of the response 
variable. The quantile regression parameter estimates the change in a specified quantile of the response variable 
produced by a one-unit change in the predictor variable (Koenker, 2005). This allows an examination of whether 
students in some quantiles based on number of days absent were more affected by basic messaging than 
students in other quantiles. The magnitude of the effect for different quantiles is reflected in the change in size of 
the regression coefficient for each quantile. This analysis defined the quantiles in terms of deciles in the 
distribution of days absent, and the analytical model included the same covariates as the main impact analyses. 
The following model was used for the quantile regression: 
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E.6. y,= B) + B?Basic_ben; + B> Basic_con; + B} X;+ BPS; + &? 


Where O < p <1 indicates the proportion of the population having scores below the quantile at p; y; is the 
response variable (number of days absent); Basic_ben; and Basic_con;, are indicators for the two basic messaging 
approaches; X; is a vector of baseline student characteristics variables, including race (African American, Native 
American, Asian, Hispanic, multiple races), gender, disability status, English learner status, whether or not a 
family had a working cell phone at baseline, and the baseline indicator for a history of high absences; and S; is a 
vector of school indicators. The quantile regression was estimated using the R package quantreg. 


Exhibits E.8b and E.8c provide a graphical presentation of the quantile regression results. Exhibit E.8b shows the 
estimated effect of benefits-framed basic messaging, and Exhibit E.8c shows the same plot for consequences- 
framed basic messaging. Each plot shows the 95 percent confidence intervals in the shaded area, estimated from 
bootstrapped standard errors. For reference, the solid red line shows the overall estimated impact for the full 
distribution (as shown in Appendix C), with the dotted lines representing the 95 percent confidence interval for 
the estimate. *? Each of these plots shows that there is no effect of assigning one of the basic text messaging 
approaches for those that were absent for a small number of days, but there is a significant effect for those with 
more fall absences. 


Exhibit E.8b. Estimated effects of benefits-framed basic messaging on number of days absent compared 
to no-messaging control, by decile 


-0.1 0.0 0.1 


Difference vs 
No-Messaging Control 
-0.4 -0.3 -0.2 


-0.5 


-0.6 


0.2 0.4 0.6 0.8 
Quantile 


SOURCE: District administrative records 
NOTES: Sample size = 23,133 students in the analytic sample. 


” The difference in the overall estimated impact of benefits-framed messaging in Exhibit E.8b (-0.22) compared to Exhibits 
C.2a and E.8d is due to rounding. 
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Exhibit E.8c. Estimated effects of consequences-framed basic messaging on number of days absent 
compared to no-messaging control, by decile 


a 
3 


S 
So 


Difference vs 
No-Messaging Control 
0.3 -0.2 -0.1 


-0.4 


-0.5 


0.2 0.4 0.6 0.8 
Quantile 


SOURCE: District administrative records. 
NOTES: Sample size = 23,133 students in the analytic sample. 


The exact point estimates of the effects of the basic messaging approaches compared to no-messaging are shown 
in Exhibit E.8d for each decile. Like the graphical presentations, the results in Exhibit E.8d show that the effect 
begins to appear for deciles in the middle of the distribution, where the number of days absent started to rise.** 
This means that the average impact of the text messaging was driven by an effect on attendance for students who 
had more absences, but not only on the students who had the most absences or who were closest to the 8 percent 
absence threshold used in this study to assign intensified messaging. This makes sense, as mechanically, basic 
messaging would have at most a limited effect for students in the lowest three deciles, for whom the number of 
days absent was already zero or close to zero. 


4 The results for the last decile (90.1-100.0%) are excluded from Exhibits E.8b and E.8c because the bootstrapped standard 
errors indicate that the point estimate is very imprecise. Including these results would require major changes to the scale, 
which would make the graphs much more difficult to read. 
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Exhibit E.8d. Estimated effects of basic messaging on number of days absent compared to no- 
messaging, by decile 


Benefits- or consequences- Benefits-framed basic vs. Consequences-framed basic 
framed basic vs. control control vs. control 

0-10.0% 0 O O 
10.1-20.0% O O O 
20.1-30.0% -0.01 -0.01 -0.01 
30.1-40.0% O O O 
40.1-50.0% O 0) 0) 
50.1-60.0% -0.06 -0.06 -0.06 
60.1-70.0% -0.25 -0.25 -0.25 
70.1-80.0% -0.33 -0.31 -0.27 
80.1-90.0% -0.36 -0.44 -0.35 
90.1-100.0% -1.0 -1.0 -1.0 

All Students -0.20 -0.20 -0.20 


SOURCE: District administrative records. 
NOTES: Sample size = 23,133 students in the analytic sample. 
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